Tuesday, August 4, 2020

Unicode migration is harder as everybody told you - if you are old.

Harder? Why harder? Everybody is telling you - it's easier as you think.

With every new Delphi Version there is this kind of movement:

Do your Unicode conversion today and use our new Rad-Studio-Version to do your work even better as before. ( Combined often with a special offer )

I fully understand this and, to be honest, this is true and it's also a good idea. Using old Delphi-Versions is horrible. Yes, the old versions are a little bit faster and perhaps the IDE is a little bit more stable. But this is a bad tradeoff because you miss all the new stuff that makes your life so much easier.

Ok, if everything is so fine and easy, why this blog post?

If you have old source code in your legacy application and the oldest unit is created after the year 20xx, you probably have no problems. Because you've used the Rad approach and have clicked DB Components on your form direct bound to your database. Perhaps you used the BDE, but converting this to Firedac is not so hard.

So far so good... Buy the latest Delphi Version to go ahead...

But if you're old and you really learned how to code in the old days with Turbo-Pascal, you are not using DB-Components at all. 

You are using a Record to hold your data. The records had to be aligned by byte or had to be marked as Packed Record.

You store a date in 3 bytes and of course, you had to use typed string to match your needs. 

In these days you had this kind of declaration because every byte counts:

   Str6  = String[ 6];
   Str26 = String[26];
   Str40 = String[40];
   Str80 = String[80];

   TAddress = Packed Record
     FirstName : Str26;
     LastName  : Str26;
     Street    : Str80;
     Zip       : Str6
     Town      : Str80;

  Address : TAddress;

With this definition, you just wrote an address to disk by using blockwrite.

This kind of code works with TP 1 up to Sydney. So what is the problem?

Changing these (short)Strings to a Unicode-String or even to an AnsiString, You are unable to write them to disk anymore because a long string is only a reference, not the actual data. No Problem you just have to serialize the data for read- and write to stream. Each dataset in this stream then has a different length and you have to create a jump table to find the record starting pos. This is the best point to store your data not in a flat-file anymore.

Let's assume you have managed this in your whole app...I still haven't yet.

Long before I had converted my source code to able to compile with XE, I thought it is a good idea to convert every String to an AnsiString, every Char to an AnsiChar and every PChar to an AnsiPChar. 

(At this point I was not aware, that the VCL is not working like the Windows API where nearly every function has an A and W version for Ansi and WideStrings.)

So each

Label1.Caption := Address.Firstnames;

produces a warning while compiling. The Compiler is doing the trick here to convert a short string to the Unicode-Caption-String - so it works. btw. this is called a possible solution in most whiter papers. It is not - Yes it compiles and yes it runs, but the warnings are the problem, we will see later.

Again where is the problem? I call it the "Var Param Problem". 

Imagine you have the record from above. Then you probably have some methods to deal with this data structure, like:

Function MakeName(var Firstname, LastName : Str26) : Str80;

Using var was a good idea and not to copy 54 bytes on the stack.

Perhaps the MakeName function calls another method inside:

Function Crunch(var S : Str26) : Str26;
  While (S[0] > #0) and (S[byte(S[0]) = ' ') do
    S[0] := Chr(Byte(S[0]) - );

  Crunch := S;

Function EmptyName(var S : Str26) : boolean;
  Emptyname := (Crunch(s) = '');  

Function MakeName(var Firstname, LastName : Str26) : Str80;
  if EmptyName(Firstname) 
    then MakeName := crunch(LastName)
    else MakeName := crunch(LastName) +', '+crunch(Firstname);

Yes, if you did not know this - before RESULT was the way to go, assigning the value to the function name was the way you had to write functions!

There where more stupid things we had done in the old days.


Kills the reference and produces a memory leak if you have a long string in the record.

Move(Address, OldAddress, sizeof(Address));

Moves the reference and doesn't create a copy of it.

With hundreds of Methods, using short or special short string types as var params all over your source code, there are no easy ways to convert all of this step by step to a different string type. Once you have changed one function that is called by nearly everybody you get this snowball effect of compiler errors.

Yes, most of the conversion beside the "Var Param Problem" is done by the compiler. The problem is: You will get thousands of warnings and you can not ignore them, because some of them you have to deal with. 

The hard task is: Find the 500 or more problems out of 60.000 warnings. I've started with XE2 and by ignoring most of the "probably lost" warnings because you assign a string to a short string, I'm down to ~7500 warnings.

But here are some good news. With 10.4 we get new managed records that can have an assign and copy method. This is the first time there is a possibility to handle long strings in records the right way by creating a new instance on copy.

It would be so perfect if we had a compiler switch to turn of Unicode in XEx - sorry just dreaming.

While refactoring the code to XE - of course, everything has to be binary matching 1:1, so every not converted data-structure still written to disk is still the same data. By using any sourcecode repository you are able to maintain both trees (Unicode and none Unicode) for a small-time until the conflicts are too much. So the only way to go is to be able to make changes to your non-unicode source, which produces the same results with fewer warnings if you take the XEx compiler. By going this way, you want as little as possible IFDEF's in your source, because you've already a big pile of code to refactor and IFDEF's in the long term makes it more unreadable.

So, how to find a way to go?

It would be perfect if I could keep the short strings in my records as long as possible. Outside of these records, everything could be "normal" string.

- impossible

Everything is calling everybody and nearly every unit is linked to every unit - not always in a direct way, but over some unit in most cases.

Side note: Don't watch Uncle Bob's Clean Code stuff, because after that you will hate your work of the past 35 years much more you already do. (were young we needed the money)

No this is wrong: EVERY Developer should watch this Session 1-6 from Uncle Bob.

Stop reading here and click on this link and after that - if you still think, you do not need to write unit-tests. Sorry, in this case, I can't help you at all. (The only possibility is: You have skipped some parts of the videos).

If I had a trustable coverage of unit tests already in place in my legacy app, EVERYTHING would be so much easier and no fear of changing some of the old methods and dependencies.

And now? I've started a different approach - by using my source-code-tokenizer from my sourcecode formatted project, I was able to find all dependencies in all my Unit-Uses and removed ~400 Units that are not needed anymore in "this" unit. Not bad, but not enough.

I plan to restructure my dependencies with a brute force or NN network - we will see if this could work. The next thing would be a "path through the source" finder, to convert all Var-params from short to string where possible.

So a long way to go and in the end, we are on VCL 32Bit. 64Bit would be the next step. If everybody is going the ARM-Ways perhaps we have to convert to FMX some time in the future.

To get something done, I'm going to ways, first decoupling and writing unit tests, and second try to get rid of the short string or at least the none critical warnings.

And of course everything just in my spare time. I wish I had...

If you have any good ideas - I like to read them in the comments.

So long... Happy coding and watch Uncle Bob!

Friday, July 24, 2020

My Fight for FMX!

If you know me or if you read my blog, you probably know, that I stand 100% behind FMX.
Well and I'm "into FMX" since XE2, but let's say with XE6 or better XE8 it was really useable for every kind of development, desktop, and mobile.

For many years I felt a little bit lonely in this field of development, but since XE10 the club of FMXer is growing faster and faster.

Finally one of the VCL Component developers is using the FMX road, too!

So please clap your hands and welcome DevExpress to the club.

Fun fact: The blogpost about the new DevExpress FMX Grid CTP is posted at/from the VCL Team with - still the VCL logo.

They also promise to include "every single VCL product" in their FMX offering... I think this is a term for: "We will provide an FMX Version of all our components".

So to all haters who haven't taken FMX seriously and/or my love for it for years, I would like to answer with a quote from the blogpost of DevExpress: 

"If everyone moves to FireMonkey, we’ll be sure to follow."

Have a nice weekend!

PS.: If you want to start with FMX, don't forget to buy my Firemonkey Development Kit.

Saturday, June 13, 2020

CodeRage 2020 - Quickinfo...

Oh boy... Why has it been always so hard to do the easy stuff?

My Idea was - just make to sessions, do some advertising on a sublevel and I'm done...

But for session one, I had to explain so many things and create "some" screen-shots. 
Normally a 45 min. video takes ~12h to create. Sometimes a little more, if I have to develop the examples first.

I'm getting faster on "the creating video task", but this time...

The first 10 minutes cost me 4 days. Perhaps I'm doing something wrong.

And session 2?

My estimates where "double the time" from session one... I did not even start the creation. 

It's time to speed up.

I'm also getting faster for the "two audio tracks" thing. Synchronizing the English and German audio is not a big deal anymore. (OK, it still cost 2,5x the time comparing to a single audio track, but faster than doing the disturbing subtitles)

I hope you will like the two sessions and the work was not in vain or different than you expected from the title.

So don't miss it on the 2nd of July! (English versions of my videos are online while streaming the German Versions).

PS.: Of course if you're are a user of my FDK - all the stuff that is shown in the video doesn't need to interest you, because many more sophisticated routines are included in my framework. Perhaps take a look anyway, perhaps just for fun.

Monday, May 25, 2020

German CodeRage 2020

The German CodeRage will be live on the 2nd of Juli. I will present two sessions.

  1. 09:00 UTC Threads and Queues (11:00 MEST)
    How can I accelerate my application by using Queues and Threads to execute some workloads in the background? My session will show some examples of how to include this kind of asynchronous data processing in your app.
  2. 16:00 UTC SQLite in Threads (18:00 MEST)
    How can I use threads to access an SQLite-Database? Spoiler: you can use the technics from session one.
If you want to see my session live at the CodeRage event, you should register and you can also be part of the Q&A session.

If you prefer to watch the session in the English language, it will be online on my Youtube Channel at the same time!

So stay tuned and please subscribe to my channel perhaps you will find some teaser ;-)!

Monday, May 11, 2020

Delphi 10.4 Sydney - #Delphi104 - #ComingSoon

Hello, my friends!

Yes, I've got permission to blog about the upcoming new version of Delphi 10.4 Sydney!.

First I want to answer the most urgent question for all FMX-Mobile developers:

What about ARC?

It's gone, it's history, I hope I will never see any kind of ARC again!

Next on my list is Metal... No more OpenGL-(ES) on iOS.

It has a nice new feature: You can set your framerate, e.g. fix to 60FPS or only refresh the screen if something has changed. I haven't done any longterm tests with this setting, but I assume that this will expand your battery life.

With both - NO ARC and Metal my iOS app is flying! Not only by numbers, you really feel the new performance. 

The other thing: The new Managed Records - as Marco Cantu has already blogged about it. Please follow the link!

Why are these records so interesting? Well, I have a 34 years old huge app grown from TP to Delphi. In these App, we're using mostly records for everything. At the moment we can only use short-string. 


Imagine a record with an ansi- or unicode-string like:

TFoo = Record
  Name : Ansistring;

Every procedure that want's to use a TFoo instance doing:

Procedure Bar(Var aFoo : TFoo);
  Buff : TFoo;
  Buff := aFoo;
  aFoo.Name := 'Othername';
  aFoo := Buff;

Because you only copy the reference of Name, changing aFoo.Name also changes Buff.Name. So the Buffer is not working at all!

With the new Records you have a copy method, where you can explicitly call Setlength to create a copy of the string.

The first time in history the migration from D2007 to 10.4 will saves a lot of work. We will see - still a long way.

BTW: Now it's a good time to renew your subscription. Please click on the banner below.

Happy coding with this #ComingSoon new Version of  #Delphi104!

Saturday, May 9, 2020

The database on network file problem!

Perhaps you're lucky and you're using a database server for your application. So any user is able to have full access to the server anytime. Or your application will not run on a network and you can easily use a simple file-based database like SQLite.

If not, welcome to the club of developers using no databases or any hacking tricks around the shared access problem.

There are some implementations out there in the wild with different approaches to overcome the problem.

You may ask: Why you don't just install a local database server?

Of course I could install a firebird server or the free MS-SQL Express server, but in these cases the PC must always be switched on, so that he is accessible in the network to play the server role. As always the easy solution is not possible.

Many of my clients have a really simple network using the "Fritzbox" as the router and up to tree PC connected to it. To be able to run our software on any of these PCs, without having to switch on the "Server-PC" each time, there is a NAS connected to the Fritzbox with all the data.

Shared access is handled by the file system. Yes, this is a working solution, no question! But as every application and also the stored data is always growing, there is a point where a real database would solve many problems.

Googleing for SQLite and Network you will find some implementation. uSQLiteServer, SQL4Sockets, SQLite ODBC Driver, or SQLiteDBMS. And you also find an easy protocol for handling these calls (TechFell).

Without going too deep into the research, everybody is using some kind of TCP/IP / Socket handler to restrict the access to the "database".

So why restrict to some cheap interface that can only handle the easy stuff?

Let's collect our needs: We want:
  • locking
  • an easy to use interface
  • threadsafe would be perfect
  • perhaps asynchrony access
  • some kind of remote procedure calls
  • perhaps some kind of caching? 

This should all be possible to write in a reasonable amount of time. The caching could be a challenge, but I would love to see a 64-Bit implementation of this, that is useable from a 32-Bit application, so the always empty 14 GB of spare memory could finally be filled with something useful.

I think I will start with a nice slim socket implementation, using UDP Broadcast to find other clients or "Server". Then connect over TCP, implement a simple low lever protocol for the handshake, ping, and version checking. Perhaps a plugin system that is able to auto-reload new versions from a server.

Yes this is all doable...

Why reinvent the wheel? My answer is as always: Because my wheels are running better. Or at least I think so... 

One big problem is still in my way: Find the time for doing this.

Perhaps you would like to see me live on YouTube trying this? Or my break down, because I have underestimated the problem... In any case, please leave a comment and subscribe to my channel.

I have to travel to a planet with a lower rotation speed!