SaferPC.info

Syncaid

Syncaid is a powerful, easily-configurable Internet downloader tool, designed to be reusable, dynamic and very portable. Oh yeah, and free! Current version is 1.1.0.4.

Without a settings file, the application itself (weighing in at less than 200kb) is quite portable - but it doesn't do much without one. Running the program without parameters and without a settings file of the same name beside it will open a minimal configuration window, which provides you the ability to create a custom settings file easily.

Syncaid supports more than a dozen different settings for each Sync configuration.

On your PC source direct saveas Resource Selection & Filters after before extensions match first last limit child Troubleshooting and Information browser debug notes Advanced Features clipboard parsing extract execute overwrite assume filehippo filehippocats sourceforge microsoft includes

On the Web HTML anchors saveas extract exec Sample Configuration Files NirSoft Password Recovery Tools SysInternals Utility Collection Virus & Malware Removal Tools Various security utilities Snapshot - IMMEDIATE backups AVG installation package FileHippo Synfig creator ...and more here Contact Contact & feature requests

On your PC

Configuring a Syncaid settings file is intended to be both feature-complete as well as user-friendly. Running the application without a settings file enables you to create a simple settings file within a form.

By default all options are disabled, and must be enabled in order to have an effect. Also, the Sync group that is loaded by default is the one with the same name as the configuration file. Thus, mysettings.ini will load the [mysettings] Sync options. Sync files, Sync groups, and Sync Options are case-insensitive, though the URLs processed may be case-sensitive, depending on the server you are connecting to.

The source option

While no settings are required, Syncaid will not attempt to download anything at all if there is no source or includes. The source option has two different implementations. Used in conjunction with the direct option, it can be used to address a single file for downloading. Without the direct option enabled, Syncaid assumes the URL is to a web page, PAD file, or text file containing links to the resources you wish to download.

Examples:

The direct option

The simplest Sync file would include only 3 lines, as so:

This would cause Syncaid to download file.ext and store it in the folder beside Syncaid.

Resource Selection & Filters

The after and before options

Most pages will require a little bit of massage to get exactly what you want out of them. If a page has a consistent structure, you can use before and after to extract a specific region of the resulting html or text for parsing. Think of it as a way to eliminate all but the region of text where the important links are.

Important note! When both before and after have tokens assigned to them, the after token is parsed before the token assigned to before!

If a token is not found, any limitations it might have imposed are ignored. Thus, if the alphabet (a to z) were the value to be parsed, with the following settings, r to z would be returned:

Again using the alphabet for demonstration purposes, the following would return everything between (but excluding) j and s:

Also note that carriage returns cannot be used in before and after tokens.

The saveas option

If you are not happy with the filename that is provided by the site, parsed automatically or just want to be creative, you can use the saveas option to name a downloaded anything you wish. This option works only when the direct option is enabled.

The extensions option

The original purpose of Syncaid is to act as a synchronization tool, obtaining the most current versions of applications and installers. With that in mind, you can see how important a simple option to get only files with a certain extension could be. This option can have multiple extensions, separated by a pipe character (|) or a comma (,).

Examples:

One of the limitations of the extensions option is simply that the files you wish to match will not always be linked with the correct extension, so parsing the source cannot provide every instance of those files. In these instances, your next best option is match.

Note that type is a functional alias of extensions. Either will work, but if both are included, extensions is the preferred value.

The match option

The match option is for those situations when you need finer control over exactly what is downloaded. With this option enabled, Syncaid parses the URLs from the source, first applying before and after to limit the context range, then it tests the resulting URL code against the match regular expression pattern.

The distinction here is that in some instances only a partial URL is returned to the client to direct to a resource. For example, when a link includes only the filename:

In the above instance, the text that is tested for a regular expression match is only file.ext. While in the case of the following link the entire URL would be tested for regular expression matching:

The last option

A common requirement on sites that provide many mirrors or a series of versioned files is the need to exclude all but a single link from the resulting series. The easiest method of obtaining only the last item in one of these series is the last option, which parses the page for links matching the other requirements, but then excludes all but the last match.

The limit option

In some situations you'll want to obtain more than the first or last matching result from a set. The limit option, used in conjunction with either first or last enables you to sync that many files from the beginning or end of the matching results series.

This example would download the last three executable files:

The first option

A common requirement on sites that provide many mirrors is the need to exclude all but a single link from the resulting series. The easiest method of doing so is the first option, which parses the page for links matching the other requirements, but then excludes all but the first match.

The child option

Because of Syncaid's recursive and reiterative power, you'll find that you can chain configuration files to perform two, three or even several hundred operations in sequence. While that provides you amazing options for downloading "many" files, it can easily overwhelm the originating folder (the folder containing Syncaid), with the volume of files downloaded.

To cope with this issue, Syncaid provides the child option to store files within a child directory (subdirectory) from where the Syncaid application resides.

Assuming Syncaid is running from C:\Downloads\, the following setting would download matching files to C:\Downloads\special\.

This option would store matching files within C:\Downloads\Installers\XP\:

Troubleshooting and Information

The browser option

Some websites are very elite. They provide only certain types of browsers access to their content, so it's important that you're able to act like you're using one of those browsers. The default Syncaid User-Agent string identifies it as IE6 on Windows XP. You can use the browser option to change that to anything you want.

The debug option

If something isn't working right, you can enable the debug option in order to have the configuration file report important troubleshooting details. Open the configuration file after or during processing to see the history data.

The notes option

Have you ever gone looking for something, only to find that it's already in your hand? That's not my favorite type of experience, so I try to document what things do as I initially create it. That makes it possible to actually look back at my notes and see where my head was at the time. If a website changes their structure, filenames, or content, it may become necessary to revise the Sync configuration file.

You can take advantage of the notes field to document your Sync file, or you can manually edit your Sync files to add comments to it. Start any line with a semicolon to make a comment.

Advanced Features

Yeah, I know. Most of Syncaid is advanced...but the following options are really advanced. Use them at your own risk.

Clipboard parsing

Syncaid is designed for portability and utility. Clearly, the most portable application is one that doesn't even require a configuration to follow it around. The Syncaid clipboard parsing engine is designed specifically for those of us that might need a custom configuration, or need to share it with people that aren't tech-savvy enough to save a Synfig file.

Create the content the same way, with the only caveat being that you can't use a root section based on the file name. Since there isn't a file name, that doesn't work. You can get around this limitation by using includes option.

For an example of how to setup includes to provide a hook into a clipboard section, checkout the FileHippo Synfig creator.

The extract option

If you have 7-zip installed and in your path environment variable, enabling this option for a Sync configuration will attempt to extract all compatible compressed files to the same directories where the resource was downloaded. For example, take the following Sync configuration:

This will extract the contents of file.zip to the folder Syncaid resides within or as designated by the child setting. It will also keep a copy of the file.zip file within that folder, as a precaution in case 7-Zip is either not installed, or not within the path.

Now take this configuration:

This would extract the contents of all of the four types of compressed files found on the list at http://example.com/folder/files.htm to the folder C:\Downloads\extracted\. Pretty nifty, eh?

The execute option

I'll take a moment to remind you - use these advanced options at your own risk!

The execute option enables you to execute all downloaded files from the directory they are downloaded to, immediately after downloading. As a geek, you surely can think of situations where this will come in handy. I have. :)

The sample above downloads all exe files from http://example.com/folder/files.htm, executing them one-by-one as they are downloaded. The Sync configuration below downloads all zip and exe files from the same webpage, then extracts the zip files and executes the exe files. Note that only the exe files downloaded directly by Syncaid will be executed. It will not extract an executable from a zip file and then execute that executable.

The overwrite option

The overwrite option enables you to enable or disable overwriting existing downloaded files of the same name. By default Syncaid will always download the version at the download URL, even if it is not a newer version. Hey, the point of Syncaid is to synchronize, which means it expects to download!

There are rarely situations when the newer version of an installable application will use the exact same filename. More often than not, new versions include a version number. However, in some situations, for example, with Filehippo, you may be using a synfig that attempts to download several files and only really want the newer stuff. Re-downloading pre-existing files, especially when they run 20mb or even 200mb each, can waste quite a bit of time and bandwidth.

The overwrite option attempts to resolve that issue. Set the overwrite option to 0 (zero) and it will skip downloading and replacing files that would be named the same thing.

The assume option

The assume option is used primarily with the overwrite option, to provide a means of assigning an extension to downloaded files to ensure they are properly matched with an existing file.

This option can have multiple values, separated by a pipe character (|) or comma (,). If the multiple value syntax is assumed, a file of the same name and any of the matching types will effect a skip if the overwrite option is disabled.

The filehippo option

Perhaps one of the most effective software distribution sites for keeping application versions and download sources up-to-date is FileHippo.com. The beauty of FileHippo's sharing through Syncaid, is that you need only know the key to an application to enable it as a Sync. Enter the keys (multiple can be separated by commas), using the format below:

or

The filehippocats option

The filehippo option above demonstrates how to sync a single package using FileHippo.com as the versioning source. The filehippocats option is very similar, with the difference being that instead of selecting a single application or package to download you are selecting an entire category. Syncaid will check SaferPC.info for the current listing, and then enqueue downloads for each of the associated packages. Enter the category codes (multiple can be separated by commas), using the format below:

or

Or use the FileHippo Synfig creator to generate a configuration file for storage OR for use with the clipboard parsing functionality.

The sourceforge option

SourceForge.net is the most popular open-source collaboration website. Among the far more than 200,000 open source projects on SourceForge are some of the most popular applications, such as phpMyAdmin, 7-Zip, FileZilla, VirtualDub, Notepad++ and others. Syncaid takes advantage of the well-defined structures on SourceForge to enable Sync of one or more packages, based on the project name and, optionally, a regular expression pattern of the packages you wish to synchronize. Note that by default the package-matching pattern will match only exe and msi packages. Enter the keys (multiple can be separated by commas), using the format below:

or

Pattern-matching for release packages is accomplished by using a colon to separate the key from the pattern. The most recent packages matching the pattern will be Sync'ed. The following pattern downloads only release versions (not beta) of DQSD:

This one downloads release versions (not beta) of DQSD as well as the English-only phpMyAdmin "zip" package.

The microsoft option

Perhaps one of the most difficult sites to navigate and locate content on is Microsoft.com. Instead of a friendly name, download packages at the Microsoft Download Center are tagged with a GUID, a 36-character alpha-numeric character that has all but no meaning. Remembering which package a GUID is assigned to is near-impossible, and Synchronizing the most current versions of MS Download packages for offline installation has been an act of insanity. While Syncaid won't make it all better, it will enable you to setup a Synfig specifically to those packages you wish to monitor and Sync only updated versions. You'll have to find out the GUID in advance from the Microsoft Download Center, but then you can use the GUID in your Synfig to update any packages that do not require Windows Validation. Enter the keys (multiple can be separated by commas), using the format below:

or

The includes option(s)

Up to this point, I've treated you to the simplistic single-configuration-within-a-Sync-configuration-file approach. includes are the means by which you can chain multiple Sync configurations and/or multiple Sync configurations files together. Don't worry, it keeps track of what configurations were previously loaded, so it's safe to "loop" them without bringing the entire Internet to a screeching halt. ;)

Includes are processed within each configuration file whenever a configuration file is loaded, unless the specific include or Sync section has already been processed. A single configuration file can include multiple Sync headings, as long as they're named differently. Sections named anything other than the name of the configuration file are not loaded unless they're called by an include directive. Here's an example:

This Sync file downloads the exe files from http://example.com/folder/installer.htm; then downloads the zip and cab files from http://example.com/folder/updates.htm, extracting each of them.

Here's a series of four Sync files, the first named 3Stooges.ini

This is Larry.ini

This is Moe.ini

This is Curly.ini

You can activate any individual Sync file by dragging and dropping it onto Syncaid. You can also rename the Syncaid executable to another filename, such as 3stooges.exe in order to change the default associated configuration files.

With the above Sync files you could drag and drop Moe.ini onto Syncaid - and nothing will happen. The reason is that there isn't a Moe section within the Moe.ini file, and there isn't an includes section either.

However, if you were to drag Curly.ini onto Syncaid, it would download and extract the "poke-y" rar series from the poke section (which was instantiated by the includes), then it would load Fingers and Windmill sections from Moe.ini, in succession - renaming the fingers.txt file to a batch file, and extracting the zip files from Windmill to a child directory named air. By instantiating Curly.ini first, the Larry.ini file wasn't touched, nor was the 3Stooges.ini file.

If we instantiate 3Stooges.ini first, Larry.ini, Moe.ini and Curly.ini are each processed in succession, though Moe.ini still has no effect directly, until called by includes in the Curly.ini file.

includes are clearly very powerful, but with great power comes great responsibility. It's important that you include a numeric includes values within the includes block to indicate how many includes you want to include. Then, for each include you need to name the value include# where # is the incremental numeric value of the specific include. They should be in the form of include1, include2, include3 and so on. Failure to properly label your includes will result in either nothing happening, or nothing happening. In either case, you won't see very much happening. ;)

---

On the Web

You do not need to do anything special in order to enable Syncaid to work with your site, but you can use some options on anchor tags in order to facilitate a more user-friendly experience.

The most important thing for your anchors is to have a useful file name. In most URLs the filename is the right-most portion of an address, following the right-most slash (/). For example, for the URL http://example.com/folder/file.ext the filename is file.ext. Syncaid will correctly detect this filename from a link to that resource, and name the file appropriately when saving it to your system.

However, some sites use click-tracking or other funky URLs to represent standard files. In these instances, a URL may either contain invalid or unfriendly characters, or simply not include a filename at all. For example, http://example.com/folder/ or http://example.com/folder/redirect.ext?query=file. In some of these instances Syncaid will correctly parse the filename on its own, but it's safer to link to the resource using a descriptive attribute, which can be parsed for the purpose of obtaining the correct filename.

To do this, you need only include a rel attribute for your anchors, with a saveas property wrapping the filename. Here's an example anchor:

Syncaid will see this URL and download the resource addressed at the href as spiffy.jpg.

To do this, you need only include a rel attribute for your anchors, with an extract[1] property (the "1" is required). Here's an example anchor:

Syncaid will see this URL and download the resource addressed at the href, then extract the compressed file to the same directory. Note that the extract functionality requires 7-zip to be installed and within the system path variable.

To do this, you need only include a rel attribute for your anchors, with an exec[1] property (the "1" is required). Here's an example anchor:

Syncaid will see this URL and download the resource addressed at the href, then execute (run) it.

---

How to contact me

If you like Syncaid, or want to see a new feature added, please drop me a line. You can reach me by email, for questions, concerns, feature requests and so on. I am a very busy man, though, so I cannot guarantee that a feature requested will be added, nor that I'll respond to your message. I will try, of course.