Changes for page Import Tool

Summary

• Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Author

...	...	@@ -1,1 +1,1 @@
1		-XWiki.caterina
	1	+XWiki.admin

Content

...	...	@@ -4,7 +4,6 @@
4	4
5	5	[[image:attach:Screen Shot 2021-10-09 at 4.33.36 PM (2) copy 9.png||thumbnail="true" height="250"]]
6	6
7		-\\
8	8
9	9	You will see a list of import jobs that are already in the system. Click Create New Import/Export job in the upper-right corner to create a new import job:
10	10
...	...	@@ -50,8 +50,16 @@
50	50
51	51	When you are done, hit Save and your Import Job will be saved into the system. It will run the next time ONEcount scans your queue for jobs, and you'll receive an e-mail notifying you when the job is completed.
52	52
53		-\\
	52	+=== **Import Job Status** ===
54	54
55		-\\
	54	+When you save a job, the web UI sets it to "Active". We then have an import scheduler that pre-flights the job and moves it into the active queue. The scheduler checks to see if the SFTP credentials work, if the file is there, if the mapping is correct, etc. If the scheduler finds a problem with the file, it will update the last run time and set it to "processed." If there is nothing wrong with the file and it is ready to go, it sets the status to "Queued". It can take anywhere from a few seconds to an hour for the scheduler to get to the job and act on it. The scheduler is what sent you the e-mail complaining about the missing headers.
56	56
57		-\\
	56	+Once the job is queued by the scheduler, the import processor starts running the import, it changes the status to "Processing." When the import processor is done, it changes the status to "processed."
	57	+
	58	+You can queue the job as many times as you want, once the import processor starts running it, the job is locked until it either finishes or until someone kills it.
	59	+
	60	+The processor can take a long time to process a file~-~-we get files of 10 or 15 million records or more that can take hours to process. So while the processor is running, it updates a database with a timestamp for that specific job. It's like a heartbeat. This just lets us know the job is a live and still running properly. We then have a monitor running that checks this database and if it doesn't see a heartbeat for an hour, it assumes the job is dead. At that point, it will kill the import job and reset its status to Active so it can run again. It then dumps a copy of the import job logs and sends the team an e-mail that the job was reset.
	61	+
	62	+Jobs can hang for a number of reasons. Most of the time it's transient, one of the servers it is talking to has a problem... there's various reasons. It doesn't happen too frequently, and when it does, the job processes correctly the second time and the customer doesn't notice. This doesn't happen if a job takes a long time, only if it stops processing for some reason.
	63	+
	64	+