Sync Server intermittently not synching changes to remotes

[LEsposito]

SLX 7.2.2; SQL 2000; Sync Server uses sequencing but ensuing syncs examined without apparent incident

SalesLogix has touted the Sync Server as one of its rock solid applications and I’ve echoed those sentiments over our 9 years using SalesLogix whenever concerns arose with any of our 55 remote users. But yesterday, when a user identified an apparent glitch, I was able to confirm using TrnViewer that an administrator update that should have gone to this user and the 54 others was indeed not sent. Furthermore, a review of the Sync Log suggested a potentially far more sinister failure which we have not yet confirmed. That is why I’m reaching out to this community (having given up long ago on ITToolbox which has gotten far too commercial and controlling of my session with its pop-ups ).

We run our Sync Server every 2 hours and it typically runs for 1 to 2 minutes, occasionally spiking during sync storms brought on by various causes including seemingly innocent merges of a small Accounts into one of our largest Accounts, a worthy topic in itself but not the issue of this post.

On Feb 18, 2010 at 14:11 our Sync Server completed a seemingly routine sync but without a specific update that a user was expecting and should have received. In fact, this 2:10 PM run, which typically generates hundreds of updates, only generated the same 3 updates to 7 of our remote users and none to our other 48 remote users. Since there were 87 workgroup files processed in that usually busy 2 hour window, it’s hard to believe that more updates weren’t synced out. The Sync Log (below) provided no errors or signs of foul play.

Is there is anyone who believes they may have experienced something similar or has any insights on how to track this down???

My hunch is one of 2 issues:
1. The physical server was running out of memory, a problem we run into every now and then, not because of SalesLogix, but because of some questionable resource administration. I have always wondered if this caused a synching problem that was somehow eluding the Sync Logs.
2. Something processed early in those workgroup logs caused a premature exit from a coding module without generating any error message. Here’s the complete log with recurring entries summarized in square brackets [] and names changed in square brackets [] to protect the innocent:

Sync Server Log

[02/18/2010 2:10:18 PM] Closing current log file.
[02/18/2010 2:10:18 PM] Refreshing DataDictionary.

[02/18/2010 2:10:23 PM] Transferring files...
[02/18/2010 2:10:23 PM]          Transferring files from profile RAS.
[02/18/2010 2:10:28 PM]   File transfer completed.

[02/18/2010 2:10:28 PM]      Apply process beginning...
[02/18/2010 2:10:28 PM]      Using conflict resolution scheme :
[02/18/2010 2:10:28 PM]            Remote user wins, Owner wins, Most recent change wins.
[02/18/2010 2:10:28 PM]      Importing workgroup changes...
[02/18/2010 2:10:28 PM]           Importing workgroup file MKRV-A00KQWG.8JH3...
              [Imported 86 additional workgroup files]
[02/18/2010 2:10:40 PM]      Getting files from: F:\SLXLogs\Infiles\*.8JH3
[02/18/2010 2:10:40 PM]      Moving file attachments...
[02/18/2010 2:10:41 PM]      Placing remote task files...
[02/18/2010 2:10:41 PM]      Sorting and sequencing files...
[02/18/2010 2:10:41 PM]      No new files to apply to database.
[02/18/2010 2:10:41 PM]      Process pending attachments begin.
[02/18/2010 2:10:42 PM]      Process pending attachments complete.
[02/18/2010 2:10:42 PM]           3 changes sent to [Team A User 1]
[02/18/2010 2:10:42 PM]           3 changes sent to [Team A User 2]
[02/18/2010 2:10:42 PM]           3 changes sent to [Team A User 3]
[02/18/2010 2:10:42 PM]           3 changes sent to [Team A User 4]
[02/18/2010 2:10:42 PM]           3 changes sent to [Team A User 5]
[02/18/2010 2:10:42 PM]           3 changes sent to [Team A User 6]
[02/18/2010 2:10:42 PM]           3 changes sent to [Team A User 7]
               [I didn’t remove any lines from this section – that’s the full list of changes]

[02/18/2010 2:10:43 PM]      Subscription process beginning...
[02/18/2010 2:10:43 PM]           Loading account list...
[02/18/2010 2:10:43 PM]      Running subscription rules...
[02/18/2010 2:10:43 PM]        Running subscription rules for [Team A User 1]
               [Repeated for the remaining 54 remote users]
[02/18/2010 2:11:02 PM]      Sending new accounts to remotes...
               [Frustrates me that no additional info ever provided here since 7x]
               [Yet another potential topic of discussion but again not for this post]
[02/18/2010 2:11:17 PM]      Removing accounts from subscription for remotes...
[02/18/2010 2:11:17 PM]           No accounts removed from [Team A User 1]
               [Repeated for remaining 54 remote users since no subscription updates]
[02/18/2010 2:11:17 PM]      Subscription process completed successfully.

[02/18/2010 2:11:25 PM]      Transferring files...
[02/18/2010 2:11:25 PM]           Transferring files to profile RAS.
[02/18/2010 2:11:25 PM]      File transfer completed.

[02/18/2010 2:11:25 PM]      Running any pending agents...
[02/18/2010 2:11:25 PM]      No agents were started.

[02/18/2010 2:11:25 PM]      Conflict information purged.
[02/18/2010 2:11:25 PM]      Conflict information stored.
         [Note that this is the normal termination of all our Sync Server logs]

Thanks,
Larry Esposito

P.S. Does anyone know why this editor uses a PDF-hand-like icon instead of the expected I-bar editing icon? I was forced to use the cursor keys, instead of my mouse, to attempt an edit and I lost an earlier post because my Lenovo stupidly decided to include a non-standard [Web-Back] button right above the [Cursor Left] button on the acclaimed keyboard of its tablet. And now I’m noticing that my edit screen keeps popping to the top of my post whenever the Hand Cursor is moved outside the Edit Box. Ugh! Wait, it isn’t over yet, I just went to Preview my Post and I got an authentication error and lost everything! Thank Goodness for MS Word and cut and paste! Let me see if there are any other curve balls prohibiting this post! Yup, they totally reformatted my document because of illegal HTML tags. Let me see if I can save the Document as a freakin’ text file. It never ceases to amaze me how poor technology companies are with technology, present company included! Finally, I think I got it. How do I hate Sage - let me count the ways! One million and 2, one million and 3 ... Oh No! Now they've done it again. I do not assume responsibility for this final post and surely don't expect a response. Right now it's just me against this Board!

[RJLedger]

First.. welcome to the "community".

There have been some "glitches" identified in sync (and some fixed). Do you have all the HF's for 7.2 SP2 that are sync related installed?

Message Edited by RJLedger on 02-27-2010 07:35 PM

[LEsposito]

No hfs added yet.

I did notice 3 Issues but I requested additional info from Sage since the following doesn't tell me much and what little it does, doesn't seem to apply.
1-55625 - Does not return error on failing HTTP Synchronization
1-55516 - Sync fails
1-49962 - PEND mechanism problem, Failed transaction build up and AttachTemp folder doesn't process attachments

I'm fairly certain that the issue takes place during the Sending step of the Sync Process (Transacting, Applying, Sending and Clean up as discussed in the Sage Whitepaper). Specifically during the process when the WGLogs are written to Cache, compressed into TEFs and then deleted.

Is it possible that this step terminates without generating the TEFS perhaps due to a low memory condition and moves onto the next step without generating an error?And if so, couldn't Sage enter some code to trap such an error and possibly even back up the WGLogs until successful completion is confirmed?

Thanks,
Larry Esposito

[LEsposito]

Forgot to mention 2 things:
1. Thanks for the welcome - I know I came in like gangbusters but it's been a tough few days.
2. We use Mike's E1Audit so every login generates updates (i.e. it's not like those 87 WGLogs were empty).

Thanks again,
Larry Esposito

[LEsposito]

We run our SalesLogix Server and our Sync Server on the same Windows 2000 Server VM and although SLX Administrator is also on the same server we rarely use it.

Our Disk Space free had creeped down from about 8GB to about 3GB over a one year stretch and it's possible that this tipped the scales into the unhealthy environment that resulted in the Sync Failure but we're now up to 18 GB free!

Our RAM was slightly under the recommended of 576 MB (256MB+256MB+64MB) but this hadn't been problematic for about a year and we only recently started seeing error messages which I'll try to include in a future post. We are up to 1 GB of RAM now.

Sage is investigating if brief network interruptions result in Sync Log errors since ours had none.

Basically, our concern is that the Sync Server acknowledged receipt of an 87 Workgroup Files payload and they vanished into thin air without so much as an error! While the environment above may have been suspect we had many more syncs, some, even larger, that ran before and after the 2/18 failure without incident.

Going forward, whenever a remote user says they can't see something I will now investigate the integrity of the Server's sync before I investigate the integrity of theirs. If I encounter another event like this one, I will try to post it on this thread.

Thanks,
Larry Esposito

[LEsposito]

Despite some heroic efforts, Sage has not been able to replicate our problem or even an error message. It seems the Sync Process is one resilient piece of code. I suspect our marginal RAM and declining free space and a precisely timed network hiccup all worked together to cause the failure. So we upped our RAM (to 1GB) and cleared out a lot (15 GB) of junk files and it looks like we're good to go. Sage is going to try a little longer to replicate the problem but I suspect the likelihood is low. We've had hundreds of syncs under the same environment with some more intense but most not and rarely had an issue. However, I am fairly certain, we've had more than one failing.

Special thanks to Pat from Sage for her perseverence in working with a not so patient customer,
Larry Esposito

[dennissekulic]

Hey Larry,

I would suspect that the RAM and free space scenario was your likely culprit.  We ran into this type of situation about a year ago and found that the free space on our machine was the downfall.  Before we came across this, we always assumed it was the user's PC and their constant fiddling around with their settings, etc. that caused grief for us and our users.

Once we cleared space of junk files, we were back in business.

Its funny, you exhaust all avenues thinking the issue is in the program but it ends up being the hardware!  A suggestion would be to look at the event logs of the machine when you notice the issue and see if anything alludes to a hardware issue, it might help determine an exact cause...

[LEsposito]

Thanks, Dennis. It's helpful to know that someone else has experienced an issue like ours. We have been looking at the Event Logs and they were clearly suggesting issues but it's just odd that nothing of consequence was logged when the actual problem occured. Sage is still investigating and I do plan to post the errors that we saw but I'm fairly certain we won't track down the exact cause but the added RAM and storage should keep it from happening again anytime soon.

Thanks again,
Larry

[RJLedger]

To add - it is a KI that there have been "problems" with ftp sync "hanging". In actuality.. it does not really hang - it sort of gets "lost". It has something to do with some updates from Microsoft that impacted the wininet libs that MS provides and is used by SalesLogix ftp sync. This is why there were some HF's released to replace the ftp dll SalesLogix. HTTP/HTTPS sync does NOT use the wininet libs but rather a true "tcp/ip" library that (AFAIK) is 3rd party. Sage took the tcp/ip library and "re-built" their ftp with it to deal with "hanging" issues.

We have been involved w/Sage SalesLogix on a long term test cycle for dealing w/ftp synch "hang" issues for almost 2 years now. Every once in a while we get a special build to test in our own production environment. It seems we have a set of circumstances that are really hard on ftp sync. For one thing, our ftp server is provided by our ISP. We have a large bandwidth pipe (10 to 30 mb download, 4 to 8 mb up). However, we can generate significant traffic to cause packet drops. This seems to cause ftp sync to go crazy. 

So you need to have the latest/greatest updates to get around the problem.

Even so, If you do not have your ftp sync setup w/multiple ftp sync profiles using different "buckets" (directories) you can still have issues. We always setup one user - one ftp profile. You could go w/more but I recommend not more than 5 per profile. If it's a remote office - always on it's own profile (remember, w/multiple sync servers, RO can only be on the 1st one).

My suggestion is to drop using ftp sync and jump over to http/https. 

[LEsposito]

We use RAS only but it's good to know. Thanks RJ!