Pimlical - AutoSync

Overview

Pimlical AutoSync makes it possible to keep local Pimlical calendars on multiple devices in sync automatically. It does not replace, nor does it interfere with the prior DirectSync feature which provides for the controlled sync between a desktop and an Android device. AutoSync stores a continuously updated version of the calendar on a cloud-based server (currently, only Dropbox is supported). Syncing can be set to occur automatically at a preset interval, or can be manually invoked at any time.

Like DirectSync, AutoSync only syncs local databases. Android calendars, Android contacts are untouched by these processes as google provides its own software that syncs Android calendars to Google calendar, and Android contacts to Gmail contacts.

AutoSync is a new feature in Pimlical, re-introduced in V-3.7.x and it is an alternate (not a replacement) sync method for DirectSync (which supports off-cloud syncing).

Setup

To use a cloud-based server such as DropBox, the first step is to setup an account on Dropbox as all Pimlical users that will be syncing their calendars together must have access to the same account. Since Dropbox provides free accounts, you may want to set up a new account just for that purpose. At the current time, Dropbox (like many other cloud-based server solutions) offers free accounts with limited storage. The Pimlical files are so small, however, that even the smallest accounts (usually 2Gb or so) are more than sufficient to handle the storage of the PIM data and all the backups etc.

After setting up an account, invoke AutoSync manually on the desktop or Android device (Menu | Files | AutoSync in P/D or menu | AutoSync in P/A). The first time this access occurs, a web browser will be launched asking for confirmation of the access to the dropbox account. For P/D, you will need to confirm authorized access (Pimlical/Android will exit AutoSync to allow authorization to take place).  After responding positively, the sync will proceed in Pimlical/Desktop. In Pimlical/Android you will need to invoke AutoSync a second time.

Pimlical uses this folder by default: Apps\PimlicalApp\PimlicalAutoSync for the PIM data. The Apps folder is created automatically by Dropbox. The PimlicalApp folder is created automatically when Pimlical first logs into the Dropbox server (PimlicalApp being the name of the application). So the first part of that path, Apps\PimlicalApp\, is set by dropbox, not Pimlical, and can not be changed. The preference AutoSyncPath just provides the server name (Dropbox) and the remainder of the path below, so normally, the path in the preference AutoSyncFilePath becomes: CloudRail\Dropbox\PimlicalAutoSync. You should not change this unless you have some compelling reason to do so. In future releases, you will be able to select other cloud-based servers by replacing 'Dropbox' with the name of the cloud-based server.

You can select Files in the left side-panel in Dropbox to see these folders and the files within them.

If you get any kind of error initially, you may need to manually create the PimlicalApp/PimlicalAutoSync folders in dropbox. Also, try resetting AutoSync authentication (P/D: Menu | Special | Reset AutoSync Authentication, P/A: Menu | Debug | Reset AutoSync).

If you invoke AutoSync manually, you will see brief displays of its progress (at bottom right in the status bar in P/D, or via 'Toasts' in P/A - 'Toasts' are brief messages that appear in the middle of the screen for a few seconds and then disappear). If AutoSync is invoked automatically by the preference setting (see below), you will only see a message indicating that AutoSync has started and when it ends (although any error messages will always be displayed). If the preference, DebugMode is set to true, however, you will see these brief displays even when AutoSync is being invoked automatically.

The options that AutoSync uses are set in preferences - see below. By default, Pimlical performs a normal synchronize operation (as opposed to an overwrite) from the local calendar, contacts and memos on the device, to the corresponding files on the server. When you sync for the first time, there are obviously no files on the server, so an overwrite is performed from the local device to the server to set the files up for first time usage. You can also set the preference AutoSyncAction to ensure that the server is initially overwritten.

The only thing that a user will typically have to setup to use AutoSync is to set the preference: AutoSyncFrequency to a value other than 0 seconds. Remember, that syncing to the server will take several minutes or more and setting a very short value will simply cause constant annoying interruptions, so if you set a value of say 30 seconds, it's not going to work satisfactorily (nor are there really any conditions under which it's really necessary to sync that rapidly). A reasonable, absolute minimum value would likely be something in the 15 minute range.

The preference TimeOfDayForAutoSync can be used to guarantee that an AutoSync has taken place by a certain time each day. For example, you have the frequency set to 8h, so it will sync three times a day. But when you get up at 8am, you want to be 100% certain that an AutoSync took place at 8am so that you are looking at the most recently synced data. 

Resolving Conflicts

There are two types of conflicts that can occur - a conflict with other devices trying to access the server at the same time to sync, and a local conflict where a sync is being performed in the background at the same time the user is modifying the database.

Server Conflicts

A locking mechanism prevents more than one device from syncing with the server at a time, and you may see a message on your device that AutoSync has been deferred for a short moment due to such activity.  A device wishing to access the server checks first to see if there is a lock file present in that folder. The lock files have a .lock extension, and a filename that identifies the device (and you can pick your own filename for each device in a preference setting on that device). If there are no lock files present, the device creates a lock file and then checks back a brief moment later to see if any other lock files have been created due to race conditions. If so, the device removes its lock file and sets a timer to perform the lock file check again in a random interval. A brief message will appear on the screen indicating that it will try AutoSync again in a few seconds. Obviously, if the Frequency of syncing is set to a very short time interval and there are a lot of devices trying to sync with the server, you may see a lot more conflicts occurring.

Local Conflicts

Local Conflicts occur when the user is updating the database when an AutoSync operation is about to start or is in progress. Before automatically launching AutoSync, Pimlical checks to see if any dialogs that can modify the database are open (i.e. Edit dialog, a dialog invoked from a selection list, etc.) and if so, will defer AutoSync for one minute. Note that if you open up the Edit Dialog and then leave it open, AutoSync will not run. In general it is never a good idea to leave a dialog that can modify the database open for an extended period of time, so it's a good practice to avoid doing that.

Pimlical will attempt AutoSync at times where a dialog is open that can not directly modify the database - for example, if an Event Selection List is being displayed, it is safe for AutoSync to proceed, although the selection list itself will not be refreshed with any potentially new information from the sync until the user returns to the main view.

Synchronization Conflicts

These should never occur provided that you set the preference LatestUpdateAlwaysPrevails to its default value of True. If set to False, the sync mechanism will not attempt to resolve conflicts and will duplicate the items that were modified on both platforms since the last sync. With multiple users automatically syncing to the same server, this will of course result in what appears to be endless duplications of events, especially of floating events which are automatically updated on each individual platform at different times.

Prior to using AutoSync, it's a good idea to review those preferences that directly affect how AutoSync operates. Usually, the only preference a user would need to modify is AutoSyncFrequency as the default value of 0s (zero seconds) effectively disables AutoSync completely. Remember to avoid picking such a small time interval that Pimlical is constantly trying to sync with the server

Preference Name	Default Value	Description
LatestUpdateAlwaysPrevails	True	Automatically resolves conflicts in favor of item that was most recently modified
MaintainAuditTrail	false	when issues arise in AutoSync, they are logged to an audittrail file. Set this to true to help diagnose issues
AutoSyncFilePath	CloudRail/Dropbox/ PimlicalAutoSync/	Path to the Cloud or Local Network Server. Up to the slash is the server designation - for now only Dropbox is supported.
AutoSyncFrequency	0s	set to 0s to disable AutoSync, otherwise set to interval between syncs (avoid values much less than say 15m or so).
TimeOfDayForAutoSync	00:00	You can set this preference to a time of day to guarantee that an AutoSync has taken place no later than this time today. Leave at midnight to disable this feature. You must also have AutoSyncFrequency set to a value other than 0s in order for this preference to have effect.
AutoSyncName		You can override the default device name with anything you like, such as "MyNexus6" or "Macbook Air". If blank, Pimlical automatically assigns a unique device name
AutoSyncThreshold	5m	A timeout value for AutoSync. After this amount of time goes by, any device still maintaining a lock flag is presumed to have stalled for some reason.
DisplayMessageWhenAutosyncCompletes	True	Set to False to avoid seeing the brief message that AutoSync has completed (also suppresses initial message too). Has no affect on messages that appear when AutoSync is manually invoked by menu command.
LocalCalendarSyncThreshold	10	Specifies the lower limit on the number of seconds discrepancy between two date/time stamps to consider them to be identical (i.e. with a value of 10, if the date/time stamps differ by 10 or less seconds, they will be considered to be identical. Caters to differences in real-time clocks on different platforms
AutoSyncAction	Synchronize	Sets the Default sync operation for AutoSync. Can also be set to P/A Overwrites Server or to Server Overwrites P/A
AutoSyncOptions	Calendar,tasks, contacts,memos	Specifies which databases are to be synced.
AutoSyncNotificationVibrates	True	Set to false to turn off vibration for the notification that AutoSync is taking place (new in P/A: V-3.5.09). Only relevant for Android OS
ApptCategoriesToSync	(All)	This preference specifies which Appointment categories are to be synced to the Android device. If blank or set to (All),  all categories are synced.
TaskCategoriesToSync	(All)	This preference specifies which Task categories are to be synced to the Android device. If blank or set to (All),  all categories are synced.
ContactCategoriesToSync	(All)	This preference specifies which Contact categories are to be synced to the Android device. If blank or set to (All),  all categories are synced.
MemoCategoriesToSync	(All)	This preference specifies which Memo categories are to be synced to the Android device. If blank or set to (All),  all categories are synced.
DaysToKeepDeletedRecords	30	This preference specifies for how many days, Pimlical should keep deleted records in the database prior to deleting them. For AutoSync and DirectSync to work properly, it is very important that deleted records not be removed prior to syncing to a device that still has those records. If they get removed prematurely, the sync routine will think the record was missing and instead of deleting it on the platform being synced to, it will restore the undeleted record to the platform that no longer had the record. A value of 30 days (default) should ensure that all syncs have been completed (presuming you sync to all devices at least more frequently than once a month).

If you are seeing any issues with AutoSync, the first step is to try a manual sync as you will see more status and error messages if you invoke AutoSync manually (alternatively, if you set DebugMode to True, you will also see the messages even on a sync invoked from the timer) . Also set the preference MaintainAuditTrail to True as you will see information regarding the sync logged into the local Audit trail file (P/D: PimlicalDesktop_AuditTrail.txt, P/A: PimlicalAndroid_AuditTrail.txt) in the local Pimlical folder. Also, set the preference DebugMode to True temporarily as in debug mode, more information is written to the AuditTrail files.

Also look in the Pimlical folder for the log file which will have the filename: AutoSync.log as that will also have information about the sync process on that particular platform.

Look in the Pimlical folder for any error files (error files have a .ERROR extension and send those files to Technical Support for analysis if you see any. These are standard text files that you can view in any text editor, so look at those files first yourself as the issue may be obvious.

Finally, the server itself will have a master log file which shows when every device synced to the server.

The first step is to look at Dropbox and locate the Apps folder - is there a PimlicalApp folder inside that folder? And if you click on that, do you see the PimlicalAutoSync folder inside the PimlicalApp folder? If so, what files do you see inside the PimlicalAutoSync folder?

Sometimes when Dropbox hiccoughs, a Lock file may be left on the server (a Lock file has the extension .lock), in that case, simply delete the lock file and other devices should then sync (a device will always remove its own lock file, but will not remove the lock file of another device).

Resetting everything to start from a known good state

Sometimes, you want to be sure that you can restart from a known good state - for example, you have one platform that you KNOW is good, and you are not sure of what is in dropbox and/or syncing with other devices just doesn't seem to be working correctly. To handle this:

1. Make a backup of your good platform, so if something goes wrong, you won't lose your only good copy.
2. In Dropbox, delete PimlicalLocalCalendar.*, contacts.* and memos.* in the Apps/PimlicalApp/PimlicalAutoSync folder and delete all *.lock files (Where * means anything, so as an extension, this means delete ALL files with that name regardless of the file extension).
3. On each platform you want to re-initialize, delete all files in the PimlicalCalendars, PimlicalContacts and PimlicalMemos folders.
4. On each platform, reset AutoSync Authentication (P/D: Menu | Special, P/A: Menu | Debug).
   
5. Invoke AutoSync on your good platform and make sure there are no errors recorded.
6. Invoke AutoSync on all the other platforms that you want to initialize.
   

You can set up as many local calendar files as you wish, but only one can be viewed at any one time in Pimlical/Android and only the currently selected local calendar will be synced to the server. To switch to a different local calendar, use menu | Select Local Calendar. If you don't see this menu option, go into Menu | Preferences and select the preference: MenuCommandsAndOrder in the [commands/Functions] section and make sure that menu  item is selected for display.

A typical use of having multiple local calendars is when you have an enormous calendar database with lots of historical data that you rarely need to look at, but must be able to view when it's needed. By putting the older data in an archive calendar(s), you can keep the current local calendar compact so it reads quickly. And on those rare occasions where you need to look at the archive, having to switch to it and wait for it to read is a small penalty to pay for the much faster response you get by keeping the  current calendar relatively compact.

Why Deleted Records are kept in the database

Deleted records must be preserved in the database for at least a while in order for syncing to work properly. If you think about it for a moment, you will realize that if you remove ALL traces of Record X from Platform B, when you sync Platform A and Platform B, if Platform A has Record X, the sync routine will assume that this must be a new record that was never synced to Platform B and will promptly copy it to B as a new record. From your perspective it may look like the record you deleted just reappeared and something is wrong, but the problem was caused by removing all traces of a deleted record. Had that deleted record stayed in the database, then when syncing, the routines would see the deleted record on Platform B, observe that the date/time stamp was later, and then would delete record X from Platform A. So do not use the commands to remove deleted records in some misguided belief that it is a good thing to do! For one thing, the Pimlical databases are so compact that removing records to save space is a complete waste of effort and accomplishes nothing. Second, you can always hide deleted records from view, so there is no reason whatsoever that keeping deleted records should ever pose any issue.

Delete records must be kept for at least long enough that a sync will take place - example: if all AutoSync devices sync every other day,. then you MUST hold deleted records for at least 2 days and would set the preference: DaysToKeepDeletedRecords to 3 as a minimum value. If this preference is set to too small a value, then deleted records will mysteriously re-appear from time to time.

Last Revised:01Feb2018