Monday, October 26, 2020

Setup your Bitbucket replacement for your Mercurial repository

A long time ago the German DelphiPraxis community convinced me to abandon my previous strategy of saving my source code via zipping only and use a source code management system.

So I googled a little bit and found SVN.

After some days of getting used to it - I proudly announced: "I have switched to SVN"! And with that, the shit storm has begun.

Why aren't you using a distributed revision-control-system? Don't use this old SVN system! Uses Mercurial or Git!

So I took the first "Mercurial" and with a little help and a talk from MVP Uwe Raabe on one of the Delphi-Tage-Event,  I had to realize that this system "just" works very well and offers all the advantages I missed in SVN. ( Sorry Craig )

Pushing everything for backup to a network-driver or a USB-Stick works perfectly. So no need for the old SVN Server installation on my local server, because mercurial is file-based. But after some time I had to share my source code with other developers and here comes BitBucket into play.

The use of BitBucket was as easy as Mercurial itself. 

I've never used Git, but I was told that it was just like Mercurial. Unfortunately, this is not the case. Even the first time I used git, there were big differences and I immediately hated git. But this was not a big deal, because I could use Mercurial for my local repositories and use Git for sharing where necessary.

I know many more developers using Git that Mercurial, but I never got used to the remote and local heads for different users and the parallel branches are show differently comparing to Mercurial.

After Bitbucket broke away from Mercurial repositories and hg is no longer supported, I had to find an alternative.  After some testing, I was able to run the SCMManager on a virtual Ubuntu installation. With the online available tutorials, the installation was not complicated. Unfortunately, SSL did not work out of the box.

The SCMManager is a Java-Client and this is not so easy with the certbot for Let's Encrypt. As far as I know, you have to convert the certificates and install them into the java key store. I like the cronjob based renewal, so hand converting was not my goal.

I found a little hint, that the Nginx server could be used as a proxy server to do the https/SSL connection and leave the java installation without any changes.

Installing software on Linux with this apt-get stuff is really easy and I wish this kind of installation would also possible on Windows. One drawback often is the location or the necessary changes to config files by hand. In this case, I had to type the config file from scratch.

By adding a new A record to the nameserver I was able to run the certbot with the "--nginx" param script who not only installed the certificate but also added the necessary parameters to my handwritten config file. Well done - this kind of "service" is really nice...

With some tweaking and some reload/retries, the server was reachable over https!

Small repositories worked immediately, but large files could not be transferred and caused an hg error: "request entity too large". The reason for this problem could only be the installed proxy server because everything worked before. But also this problem could be solved by a little googling. 

So now I have my own Mercurial server. If you want to know more about this, please write this in the comments, then I will try to collect my google knowledge into another post.



Friday, October 9, 2020

How to write you own Serverside-Extention!

Serverside Extention (IIS or Apache)!

For many years I have my own webserver. In my case, this always was a Windows Server, because I want to do my serverside scripts in Delphi! Apart from that, I had no clue about a Unix/Linux Server!

Being part of the Apocalypse Coding Group opens this door and in many hours of development of our little game, I could prove, that the same ISAPI.dll could also be compiled and executed on a Ubuntu Apache Server! 

I love Delphi. 

Building a new website or web service in the past, I had to make a choice:

For the website:

1. ASP.NET (Too bad only up to D2007)
2. ISAPI.DLL

After using ISAPI.dll's for a long time I've more often used ASP.NET. The code-behind for a webpage is not locked by the server, so it was easy to update the DLL-files. This was also possible by using the well known "egg"-loader. Updating the server to a newer version suddenly prevented the loader to work and I had never the time to find the reason for this. That why I preferred ASP.NET and for the other ISAPI.dll's I always hat to start and stop my IIS-Server.

For the web service:

1. SOAP

SOAP was the way to go because it was so easy to use. Just define your Interface and use it as it is local in your application. The only drawback is the huge data that is transferred with every call. That's why I used this not in the "native" way but just to transport an encrypted, compressed binary stream.

This was the past... But as time goes by...

If your traffic is increasing and you want to do many more things with your server, there comes a time when you realize that a 2 core server with an ancient operating system is just not up to date anymore.

Virtualization is one way to go. In my case, I use Proxmox on a beefy 2 CPU, many cores,  many RAM Server with a RAID SSD drive... Nice...

This server handles two or more Windows 2019 Server and many Ubuntu Servers for small web- or microservices.

Of course, for web-services "we" are doing JSON/RESTfull web services these days. So what is needed to do all the fancy stuff? 

ISAPI.DLL's - best performance - native execution, no script interpretation. For a web-site there are two ways to go: (perhaps many more, but not for me)

1. Uses a "normal" HTML page and  AJAX background calls to get the dynamic content you need for your site.

2. Use the Webbroker implementation and deliver the content directly from your DLL. This content could also do REST-calls to get more data from the same or a different service.

Perhaps you also want some URL-Rewrite or another Server-side handling, in this case, you also need a Filter.dll.

This time I'll do it better...

With the newly installed Windows 2019-Server, I tried the simple Hello-World DLL that is included with the egg-loader. But the Server was unable to execute this little DLL. But a request to the URL just downloads the DLL. This was the same with other DLL's I just copied from the old server. (The Hello-World.DLL works without problems on the old server). After some googling. I found the "uses 32 bit - DLL's" setting. But with this setting, I got a 500 with a weird error message. Setting the error level to detailed brings more information. Spoiler: Do not google these errors! Everything you'll find is nonsense! Like: You have to disable compression or other things. Most of the ideas lead you to the Handler settings - also wrong. (of course, you have to install all the ISAPI IIS extensions, I assume you have done this). So what is the difference between the old and the new server? 

64-Bit!

OMG... Why is there a setting for 32-Bit DLL's if you have to use 64-Bit anyway? OK - Start 10.4.x, compile it to 64-Bit and try again. To bad the Loader-Version 2.0 is from 2005 and not Unicode. After some fixes - the same problems. Forgot to disable the 32-Bit setting. The Hello-Worlds.DLL brings no errors anymore but is download again with every call. The loader is "kind of " working. All log files have wide-strings. Unreadable. Again some Unicode-Fixes and recompiled the Hello-World.dll to 64-Bit...

And here we go... a nice "Hello" displayed in the browser. But the Egg-Loader is unable to update the *.run version with the *.update version. I think the IIS is still holding the DLL. (How is this possible). So I wrote an eMail to William Egge if he has any ideas... 

This error is impossible. The loader is working, the hello-world is working, a free library should also work and the IIS should have no knowledge about the site-loaded DLL. Why is the loader unable to rename the files?

OMG... Again... No rights! 

By giving the IIS-User the access rights everything is working perfectly! Of course, I wrote to William that everything is working now... Answer from Bill: "Ok Cool 😊"! Perhaps I will send him the changes or I will write a different version, that is also able to handle Filter.dll's. Not decided yet!

So here is the workflow:

1. After installing the Windows-Server and all necessary extensions.
2. Create a website!
3. Compile your WebBroker.DLL (64-Bit or matching the Server)
4. Create a directory outside of your web-page-path. (e.g. c:\API / c:\ISAPI / c:\Script)
5. Create a minimal web.config in this directory. (for debugging errorMode="Detailed")

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <handlers accessPolicy="Execute, Script" />
        <directoryBrowse enabled="false" />
        <httpErrors errorMode="Detailed" />
    </system.webServer>
</configuration>

6. Upload the DLL to this directory and rename it to WebBroker.run (or any other name).
7. Upload the loader.dll to the directory and rename it to WebBroker.dll (Same name as your DLL)
8. Add the IIS-User with access rights to this directory
9. Add an Application to your website in the IIS-Manager. Point to the ISAPI directory.

(The Name will be part of the URL) "API" -> https://www.yourdomain.com/API/WebBroker.DLL)

10. Read is not permitted and browsing is off, but anyway I would store log files outside of this directory.

With the redesign of the egg-loader, we are back in the game!

If you want to see a YouTube tutorial of this, please leave a comment. There will be a video about this as part of my Firemonkey Development Kit and the Server-Side of my JSON-Store implementation, but this is scheduled not before I've completed my #D.MVVM Framework series and more basic stuff of my FDK.

If you want a notification on new videos, please subscribe to my channel and hit the bell icon! Thank you, this helps to grow my channel!

Tuesday, September 29, 2020

Workflow and multi - binding with #D.MVVM - The Delphi MVVM Framework!

Workflow and multi binding with #D.MVVM!

In every example for MVVM, you find the three blocks diagram explaining the binding between the View, the ViewModel, and the Model.

If you dig deeper into a real-world application, you may use the 1:1:1 relation, but in most cases, you need it very differently.

1:1:1 means One View connected to one ViewModel, connected to one Model!

Let's start with the View:

Most of my views are not simple TForms but much more assembled on small subframes. The subframes are also TForms and not TFrames, because of the recurring problems with frames (in the past). But that is not the topic.

The View will be created in the composition root - or elsewhere - and then?

Every subframe must also be created and placed into the target frame of the parent view.

So far so good. Of course, this is all handled by my framework.

In some cases, every frame should have its own ViewModel. But what if not?

Well, keep this in mind and start at the other end of this chain...

We have a database where every row is just an entry of our beloved TPerson, with the normal address fields.

Part of this TPerson is a list of telephone numbers handled in a different table and perhaps also a list of bank accounts also in a different table.

As we all know - I hope - the Model is not our Database interface, the Model will get the interface to the database for local or remote storage. This could be an interface to a local database or to a RESTfull web service or something else!

Let us talk concretely for the moment:

To store a value into a database or send it as JSON you need the field-value. For the complete row and perhaps for the subtables you need a list of field-values and the field-names or at least the index position in the param list of the query.

We will name this: Place X.

Place X need the Name, Street, Zip and more... And of course, the two lists for the subtables.

To be able to get this right you have to collect the data. A good place would be the Model. Wait... The Model?

So for all the different fields and from at least 3 frames, we have to collect the data in only one Model!

To store the data, the database should do a transaction to keep the subtables in sync with the address table. This would be a nightmare if 3 Models would do this separately.

At this point, we have our View with 3 subframes (Address, Telefonlist, Bankaccountlist) and only one Model.

1(3) : ? : 1

The View is not the place to collect the data from the subframes. Should we bind the subframes to one ViewModel?

In this case, we get our example relation of 1:1:1. That would be nice, but will it fit our needs? One reason to separate our View into SubViews is the possibility to swap out a subview to a different version. To check if the subframe data is sufficient to be stored, I would like to have a simple ViewModel per frame. Let's collect:

PersonView -> PersonViewModel -> PersonModel <- Database/Rest Interface

AddressView -> AddressViewModel -> ?
TelefonView -> TelefonViewModel -> ?
BankView -> BankViewModel -> ?

The PersonViewModel has no fields (or just two buttons).

Even if we do not store data in visual controls, we must define the controls. Lets name this Place B,C,D for the three subviews, and Place A perhaps for the PersionView that only has a clear and save button.

In the PersonViewModel we have to declare a Field/Property canSave (Place A1). Inside the PersonViewModel we must be able to check the three other ViewModel so every ViewModel from the subviews should be connected to the "parent" ViewModel.

In the AddressViewModel we have to declare the Field/Properties for the Address. (Place B1) And the same for the two other ViewModels (Place C1, Place D1).

In a best-case scenario, we would skip the Models for the subviews - if not we would also have Place B2, C2 and D2.

Is MVVM slower in development because you have to type everything three times?

Normally: Yes, it is!

Don't forget we're doing MVVM to get things faster developed! 

How can we archive this?

One major point for MVVM is Testability. Perhaps we are slower in the first place, but with a huge project and a lot of Models - if we have really good test coverage - we are much faster than other projects after a short period of time. Especially if you're doing TDD. 

If you haven't seen my first two videos on YouTube, please take a look. And also please subscribe to help me grow my channel!

My idea is auto bind by name, this saves you a lot of time. The other idea is now to store field values on the flow from component to the final database.

Let's take a look at the full development cycle and for simplicity name it in Delphi terms:

Create a TForm for the subframes, name it TPerson.View
Create a TForm for the address, name it TPersonAddress.View
Create a TForm for the telephone numbers, name it TPersonTelephone.View
Create a TForm for the bank accounts, name it TPersonBank.View

In the composition.root: Register the View's and implement the create for the Forms.

Create the 4 ViewModels.

As the ViewModels are "just" classes derived from TViewModel, we only need to implement a property for all components on the TForms. This is the place where we would like to store our values. To speed up things, we don't create local fields as 

FName : String;

we create this local field as

FName : TProperty<String>;

Auto-Binding will do the rest and connects the Name : TEdit from the Form to this "property"!

Create an empty TPerson.Model as TPersonModel = Class(TModel).

At this point - if you're doing TDD - I would create a TPerson.Model.Test.pas and also a TPerson.ViewModel.Test.pas. But this is a topic for a different blog post.

The base TViewModel class has a procedure call ViewModelSet that is called if the ViewModel is set to the View.

If this event is fired, because you set the property in the composition root like:

Result := TPersonView.Create;
LViewModel := TPersonViewModel.Create;
LViewModel.Model := TPersonModel.Create(TPersonDB.Create('local'));
Result.ViewModel := LViewModel; // here

By setting the ViewModel Property at the View, the Framework is doing the auto depending subframe load. So if everything is ready to go you have the ViewModelSet method that is called to setup the multi bind.

So we have:
1(3) : 1(3) : 1 chain!

If our MainView / MainViewModel wants to create our PersonView it just calls "Show('PersonView')" as part of the Navigation Service - Bang - everything will be created and everything is wired-up for the View and the subviews.

Now - how to transfer the Data from the View to the ViewModel to the Model to the DB and even from SubViewModels? Sounds complicated!

Let's compare RAD vs. MVVM!

In a "normal" RAD App you will have to define some kind of data structure to keep your data in memory. This could be a Class or a Record with all your field-values. In our case, there will also be some kind of TList<TBank> or TArray<TTelephone>. To get your data into your data structure you can use the live bindings, to a dataset or in the old days perhaps a DBEdit to do this the direct way. Really horrible, because while viewing the data, the database connection must be kept open or copied to a mem-table. Perhaps you have done it the old way by manual setting the values to the component and back by calling a method like:

FormToData, DataToForm where you have to set every field to the component.

EditName.Text := Data.Name;
EditStreet.Text := Data.Street;

// ---

Data.Name := EditName.Text
Data.Stree := EditStreet.Text;

Even if you use the live bindings this could get messy if you have many controls on a Form, and of course, it is horrible to maintain, because all changes are in the DFM/FMX file not so easy to merge in your repository.

Designing the Form is the same for RAD and MVVM.
Instead of creating a Class to hold your data, in MVVM, you create a ViewModel. (Perhaps some tutorial says that the data is stored in the Model, but this is not my approach). Creating the ViewModel should take the same time as creating your Memory-Class. The auto-bind will be faster the every other RAD approach.

1:0 for #D.MVVM.

Autoload of depending Frames would be 2:0 for #D.MVVM, but you could use real frames so this would not count. But setting up subframes in the composition root is easier to maintain, so let's call it even and don't count this. So we are still at 1:0.

Using the ViewModel for a Unit-Test to check if the Form works as expected: 2:0 for #D.MVVM. This should count 3:0 because Unit-Test with the RAD-Form is not really an option.

So for the design part, we are faster with #D.MVVM than with the pure RAD approach. 

And this is where the problems begin: We don't want to redefine all the fields in the model and thus double our effort and lose our time advantage. We also want to save ourselves the typing work and don't want to double our source code, which we have to maintain in parallel. But in some cases, we have to convert fields to be able to write them correctly into the database. The same is with the database itself. We don't want to create a separate definition for our database by copying the field from the ViewModel. 

Perhaps you use a tool to design your database or some kind of Modelmaker stuff to create your DB and Classes! From my point of view, this is not the way to go. I want to have everything in the source to keep track of it over the source repository.

How can we archive this?

Well, one idea is to use an on-the-fly created data context that is able to receive or set the property fields in the ViewModel and also convert the data-fields while sending them from the ViewModel to the Model and after that to our database interface. If we finally have a wizard in place for creating ViewModels and Models insight the IDE, it will be a piece of cake to do this. Without a wizard perhaps we have to rebuild the structure only one more time to create a universal data-structure that is able to deal with our ViewModel, our Model, and our CRUD-Database.

Without this approach let's collect our places where we have to copy all the fields. We need a copy in the Model (A2, B2, C2, D2) and in our database (at least in the SQL statement to create the DB and in every query to read or fill the values) (A3, B3, C3, D3) and as we remember Place X for the JSON.

This sounds like overkill and throws a bad light on MVVM. 4 Places to copy all fields... Horrible

That's why I do it in a different way with my #D.MVVM framework. Create your ViewModel, use the ViewModel definition, and some attributes to define an ORM-Like Model for the database and also for the Model, define some converter methods to handle or change field or field-types. That's it!

No need to define any fields in the Model: 2:0 for #D.MVVM.
Just define converter if needed: 3:0 for #D.MVVM.
Create the database with just one definition: 4:0 for #D.MVVM.

And don't forget this is VCL and FMX. The next step is a View to ViewModel wizard to convert your legacy VCL app to VCL-MVVM or FMX. 

This could all be done with the pure #D.MVVM Framework. Combining this with my FDK - The Firemonkey Development Kit, that contains many units and functions that are also usable with VCL, provides the intuitive MVVM developer with a tool that can not only create even the most complicated application with numerous interfaces in a very short time but also keep it maintainable for many years.

So stay tuned for the next #D.MVVM video on my YouTube channel!


Tuesday, September 15, 2020

Submitting new Apps to the stores!

I do not often need to update my main App in the App Store or Play Store.

But every time I have too, it's a pain in the a**.

Don't get me wrong, upload a new version of my App to google, is done in 2 minutes and the App is nearly instant online.

My daily driver is an iPhone and I also have some iPad's, but the store with all the provisioning, certificates is a pain.

But even if you've managed to get all your certificates right, there is nearly every time a new key or entry for the *.info.plist. Most of these are well handled by the IDE. 

Then you upload your binary, well and nearly every time, because of all the changes by Apple, you get a weird error. After spending hours of googling, perhaps you find out the reason.

This time it was the multitasking orientations for the iPad.

All the keys like UISupportedInterfaceOrientations~ipad with:
  • UIInterfaceOrientationPortrait 
  • UIInterfaceOrientationLandscapeLeft
  • UIInterfaceOrientationLandscapeRight

The error was missing the keys form above found key from above.

<key>UIRequiresFullScreen</key>
<true/>

fixed the problem.

Then I saw that I've not included the 1024x1024 px icon - (because this is new).
After selecting the right icon - you have to increase the version number.

Submit and wait for the review. 

My App is in the Store since 2013 - this time they want to have a video from a physical device, showing the use of the location sensor. (Ok, this was an old  function that is no longer in use, so I disabled it)

And I've tested my App against iOS 14, to be sure that everything is still working...

App rejected because I mentioned iOS 14 in the what's new part.

So again a new Upload and a new version number! I really like my iPhone, but from the point of a developer - Android is so much easier to handle.

The new version is waiting for the review, we will see...

Monday, August 17, 2020

#D.MVVM - At what point is a framework ready for release?

Besides some bug-fixes and the typical daily stuff, my focus is still on my Delphi-MVVM-Framework. 
#D.MVVM

Why, because I want to port all my old applications to this and I promised myself not to start a new project until the framework is ready so that I don't get the idea of either using an older version of it or not using MVVM at all.

There is also a dream, that I'm able to port our 35 years old main VCL - Application from D2007 direct to 10.x - #D.MVVM - at best direct to FMX.

So, I have to make sure that I can call the framework finished, which means I also have to set the needed functions to the status "finished".

Of course, we all know: With such a project, as with any other application, there is never the status "finished". So how do I find out when I'm just "tweaking" it? Ok. first of all, I need a feature list. I would really like to have a unit test for each feature. Second I need sufficient documentation. (Oh, boy this is a breaking rule. I have no documentation at all - so far).

But what's about clean code - or at least cleaning up the code (a little bit)? Yes, this is 100% necessary. 

How do you find out, if an entry in the feature list is a really necessary feature and not some nice functionality I just want to add? I really like to read your comments about this. What are the core functions, you think must be included for a 1.0 Version of my framework?

Before we try to collect some features we should define some rules.

To implement an application with the MVVM-Pattern we want:

  1. No, or nearly non-code in our View! Except for UI stuff!
  2. Comparing a RAD-App with an MVVM-App, MVVM-App should be as easy to create.
  3. The bindings are the biggest problem, they must be as easy as possible to set up.
  4. No use of Anti-Pattern. (Some people are calling a ViewFactory already an Anti-Pattern)
  5. At best: Just create both kinds of  "Forms" and you could compile it to VCL or FMX with the same codebase.
  6. Every part of your application should be unit-testable, so no dependencies between your modules. 
  7. Don't break the layer concept.
  8. The framework should work without the need of the FDK. (I had to duplicate some stuff)
  9. D.MVVM integrates as a plugin to the FDK to benefit from all the higher-level functions.
  10. Don't use the visual live bindings.
  11. Everything should work, without clicking non-visual components or special components on a form.
  12. Naming conventions over configuration.
  13. Don't care if a C# developer thinks you are doing it wrong! 😄

Anything else?

If you know me, I really dislike clicking things "together". I like to do everything in code - except the form of course - so that every change is noticeable in the source-code repository. Perhaps I create some components at the end, to fit the needs of the RAD-Development, but that is not yet decided.

While creating a feature list, I can also create some tickets to keep track of the development process.

My D.MVVM Framework should contain:

  1. A Wizard to create a new D.MVVM Application.
  2. A Wizard to create a new View (Form or Frame).
  3. A Wizard to create a new ViewModel.
  4. A Wizard to create a new Model.
  5. A Framework (VCL & FMX) independent overload of the TForm implementation to give every Form some base functionality. (done)
  6. A central point to do all the joins or Viewchains. (done)
  7. All included VCL and FMX components should be bindable. 
  8. 3rd party components should be easy to include in the binding schema.
  9. Bindings from the View to the ViewModel should be possible without writing any line of code. (done)
  10. Bindings from the ViewModel to the Model should also be possible without any code. A value converter should be included in the data transfer event. (done)
  11. A data context should keep the data in memory to be displayed. (done, but not fully tested)
  12. Some basic services should be included. (Action-Service, Navigation-Service, Menu-Service, Dialog-Service, Time-Service) 90% done.
  13. A List-Adapter for handling huge Data in Grids, Listboxes, and Memos. (done)
  14. Public Unit-Test with 100% code coverage. (Some are done, but I have not measured the coverage)
  15. VCL and FMX demo Apps. (some are done)
  16. Documentation.
I think this is not the final version 1.0 list yet, but almost.

Again: After watching Uncle Bob's clean code lessons - I have to do some heavy refactoring...

If you have any suggestions for me, I would be very happy about your comment on this topic!

If you're reading my blog for the first time, please take a look at my Youtube Channel for my already released D.MVVM-Videos.

Please don't forget to subscribe and hit the like button to help me grow my Channel.














"New" Video about SQLite in Threads is online!

Ok, of course, this Video is not really new...


I just could not find the time to do the translation and record all the translated voice-over.

This is the second video of my German CodeRage 2020 session.

In the German language, the same sentence take sometimes twice the time to say, comparing to the English version. That's why the Videos has some "silent" parts.

This is the second time I've tried to use the Speechelo AI engine to create the English voice-over! It's not so bad and since I haven't gotten any bad feedback by using this engine to produce a video, It looks like you agree.

But I think next time I do it by my self again.

Producing an English video and after that translate it to German is much easier than the other way around.

So have fun.