today is a problem with the server: most functions are not running

---

Guy PFLIEGER MASEVAUX ALSACE France
phone: 0033973514697

Looks like all servers are now active. Are you still seeing upload problems?

---

Rosetta Moderator: Mod.Sense

Hi,

We are still getting upload problems. Status: Project back off.

Thanks.

I have two machines here with the same problem. 10 WU's stuck on uploading. They transmit between 48KB and 55KB then stop dead before going into the retry loop.

Upload problems remain.

I'll take a look. Sorry for being late on this.

I'll take a look. Sorry for being late on this.

at 17:40 pacific time here I still have 16 queued up waiting in line...

Happy Easter everyone!

Cheers,

/M

---

More WU's getting stuck on uploading.

We've been trying to troubleshoot this issue but still do not know what is causing it. Our sys admin said that UW-IT has been contacted to determine if it's a UW network issue. Sorry for any inconvenience.

Just to update. Some WU are actually uploading, but some are not.

I am not sure if this helps troubleshoot, I thought it may be worth mentioning.

Happy Easter, and Happy Holidays to everyone else.

Hey folks - Happy Easter to all.

Yesterday, in response contributors concerns about our aged SSL cipher handling on the R@H services, we made changes the webserver configurations. For reasons I don't yet understand, these changes resulted in upload timeouts and the slow collapse of project backend.

Regrettably, I had 'gone offline' yesterday evening to celebrate the season with family and was unaware of the problems until this AM [ADT]. We've reverted to previous configurations and restarted the project.

We apologize for the 'outage' and will keep monitoring the situation closely until convinced things are working.

FYI - we are in the final stages of building out a new, shiny, high-powered BOINC system for R@H. More info coming.... -KEL

[Update] It looks like there are still problems....

---

Could you modify your Server Status web page to show which of the server programs handle uploads and downloads?

Could you modify your Server Status web page to show which of the server programs handle uploads and downloads?

I just added some text:

Web servers: boinc, srv1, srv2, srv3, srv4, srv5 (upload and download servers)


boinc is load balanced among the srv web servers. The srv servers handle uploads and downloads.

I have seen so far one WU that is stuck, though I can't remotely check all the hosts that are running R@H.
On that one host where I noticed this since Friday, other WUs are uploading fine.
And the one that gets stuck is trying to upload but as far as I watched some forced upload retries, it craps out at various amounts of data, between 3KB and 32KB, out of 739.36KB.

The WU in question is https://boinc.bakerlab.org/rosetta/workunit.php?wuid=820755819.

I now see that there are actually a few more WUs from the same date send that should have all been returned by now, on other hosts as well...
EDIT: Actually, I just checked and there are exactly 3 more WUs from the same data (one is early morning the next day, the 3/12) on one other host, a laptop that I can't check remotely. However, that same laptop has successfully received and returned WU send after 3/11, 3/12...

As other WUs are uploading just fine, even on the same machine, I can not think of a reason as to why a networking issue at UW should be causing this... :?

Ralf

I've got 12 queued that are now past the due date.
And another 14 waiting that haven't gone past the due date yet, but
more are dropping off all the time.

These are spread out on 15 different clients.

---

Pretty similar here -- realizing that the issue is back at the Rosetta site, since iis not happening with other projects on the same systems, I've elected to suspend processing on Rosetta units, which pushes my processing over to WorldGrid.

I periodically try to push the uploads but so far (since last week) no joy here.

Ideally the folks on the project side will figure out the problem they have at there end and resolve it.

I've got 12 queued that are now past the due date.
And another 14 waiting that haven't gone past the due date yet, but
more are dropping off all the time.

These are spread out on 15 different clients.

---

Still have WU's stuck uploading. However, a different error sometimes shows up now:

17-Apr-2017 18:06:53 | rosetta@home | [error] Error reported by file upload server: [des_DS_160_fragments_fold_SAVE_ALL_OUT_470271_1422_0_0] locked by file_upload_handler PID=4156
17-Apr-2017 18:06:53 | rosetta@home | [error] Error reported by file upload server: [tj_3_6_junc_X_DHR55_DHR55_l3_t2_t3_4_v5c_fragments_abinitio_SAVE_ALL_OUT_474351_416_0_0] locked by file_upload_handler PID=23998

Still have WU's stuck uploading. However, a different error sometimes shows up now:

17-Apr-2017 18:06:53 | rosetta@home | [error] Error reported by file upload server: [des_DS_160_fragments_fold_SAVE_ALL_OUT_470271_1422_0_0] locked by file_upload_handler PID=4156
17-Apr-2017 18:06:53 | rosetta@home | [error] Error reported by file upload server: [tj_3_6_junc_X_DHR55_DHR55_l3_t2_t3_4_v5c_fragments_abinitio_SAVE_ALL_OUT_474351_416_0_0] locked by file_upload_handler PID=23998

So where does file_upload_handler come from? Who is locking the file. The host machine or the server?

My error messages are

18/04/2017 03:38:57 | rosetta@home | Started upload of rb_04_14_73903_117148__t000__ab_robetta_IGNORE_THE_REST_480361_204_0_0
18/04/2017 03:39:19 | rosetta@home | Temporarily failed upload of rb_04_14_73903_117148__t000__ab_robetta_IGNORE_THE_REST_480361_204_0_0: transient HTTP error
18/04/2017 03:39:19 | rosetta@home | Backing off 05:44:21 on upload of rb_04_14_73903_117148__t000__ab_robetta_IGNORE_THE_REST_480361_204_0_0
18/04/2017 03:39:31 | | Project communication failed: attempting access to reference site
18/04/2017 03:39:33 | | Internet access OK - project servers may be temporarily down.

Again, the server...

---

Could you modify your Server Status web page to show which of the server programs handle uploads and downloads?

I just added some text:

Web servers: boinc, srv1, srv2, srv3, srv4, srv5 (upload and download servers)


boinc is load balanced among the srv web servers. The srv servers handle uploads and downloads.

Looks good so far, but could you also add the status of all those servers to help us tell when to expect upload and download problems?

It seems to me that the upload problem has been solved. At least, all my stuck WU's have been uploaded now.