It's easy to change the number of threads in the code, but there's no way for the user to do it right now. I'll see if maybe I can add something simple for power users. There is no harm in creating more threads than your computer has cores. The program will still run OK. You just won't get any performance benefit out of it because each core has to juggle multiple threads. In fact, if you create way too many threads you'll get worse performance because of all the juggling that goes on!
My Java is the latest, jre1.8.0_271 Actually that was me copying the C:\Program Files\Java\jre1.8.0_271 folder into a .minecraft subfolder I called Javaruntime testing that it wasn't a path issue.
Just for my general interest; is there really much more performance to gain beyond having a thread to download the data, and a thread to parse the data? You'd be risking running into data synchronisation issues if you weren't careful right?
Thanks! Documentation is extra important when you are writing general-purpose libraries! Yes, there is! Each rupee transaction page can be safely parsed in its own thread because the data on each page is not related to each other. So, with a 4-core computer, you can speed things up a lot by parsing 4 pages at once instead of 1 page at a time. However, since there's no way of telling which thread finishes first, you have to make sure you assemble the parsed data in the correct order (assuming that order is important to your use-case). The threads that parse the rupee transaction pages add their parsed data to a synchronized queue. Then, a separate thread pops from this queue to insert the data into the EMC Shopkeeper database. Having a dedicated thread perform the actual writing of the data to disk helps prevent synchronization issues. There is also the issue of network latency. After sending the HTTP GET request to the EMC website for a rupee transaction page, the program has to wait several hundred milliseconds (or longer) for the response. So, it could be doing other things while it is waiting!
Not necessarily hardware, just the amount of threads currently available. I have no experience with Java, but after some web searching, it looks like this can be obtained from public int availableProcessors().
Looks good! What is the advice for if you're, for example, also running minecraft at the same time (using 2 threads)? Should you list your amount of threads, or will it then be sad? Is it possible to dynamically find the amount of free threads and use those? One thing I would change: "Your computer has 4 cores" into "Your computer has 4 threads", as some CPUs do have hyperthreading (or similar) while others don't.
I haven't run any formal benchmarks, but my recommendation would be not to overthink it. Tweaking the number of threads based on whether you are running Minecraft or not won't make much of a difference in the end. It's inaccurate to think of threads as being "free". Threads are created by programs to perform some task. A thread "belongs" to the program that created it. There's basically no limit to how many threads a computer can have running at a time. The operating system looks at all of the running threads decides which CPU core to run each thread on. Threads are not "assigned" to a specific core--they are swapped to different cores by the operating system as they are running based on what the operating system thinks will be most efficient. To be 100% correct, the sentence would read, "This Java program has 4 logical cores available to it." When you launch a Java program, it's possible to restrict the number of cores it has access to (according to the documentation for the "availableProcessors" method). EMC Shopkeeper has access to all of your computer's cores by default. A "logical core" means it could be a physical core or a core that is created from hyperthreading. So a 4-core CPU with hyperthreading has 8 logical cores. You can see this in the Performance tab of Task Manager. When a CPU has hyperthreading, the operating system acts like it has more cores than it really has. So, when a program asks the operating system how many cores there are, it doesn't say, "There are 4 physical cores and each physical core supports hyperthreading". It just says "There are 8 cores".
It seems like we're using different words for the same thing. I used these definitions: An Intel i7 6700k has 4 cores and 8 threads. If I understand you correctly, what I called threads, is what you would call (logical) cores. Applying that to my message and your reply: If I have 8 logical cores, and 2 are being used by minecraft, wouldn't the program be sad if I told it it could use 8 logical cores - as 2 are already in use and would (I think?) have to fight with the new task. Maybe changing that to "logical cores" would then be better?
It's not MY wording. It's the industry's terminology. Programs do not experience emotions. But it would likely have to do some fighting for CPU time, yes. xD However, each download thread DOES experience a short moment of inactivity while it is waiting for EMC to send it the rupee transaction page, so this would reduce a tiny amount of the fighting. Yes, that would be more accurate.
Well yeah, I figured so too, but I think that's more the programming industry's terminology - I'm reasonably invested in the computer hardware industry, and as far as I know, it's generally called cores and threads as in "my" way. Even on Intel's website, they say the i7 6700k has 4 cores and 8 threads. Yep, I figured so, so it would be better to just list the amount of free logical cores. Would it be possibly to automatically have it read the number of available logical cores and use that? That way the user doesn't have to adjust that setting, and it will automatically adapt to more loaded and more "unloaded" systems.
Oh, yeah, I guess I was referring to the "threads" used in programming. Yeah, I guess I could. Trying to think of any negative side effects to that.
That's what I was suggesting. The downside is that the program will be using max cpu even if the user would prefer it not to. There might be ways in Windows to stop it from doing so, but I don't know them. What I mean is that if a user lets your application run in the background while doing something else, they might not appreciate it using max processing power, especially not if they're on a laptop or a desktop with a loud fan. However, you did point out that the threads have to do a bit of waiting to receive the data, and that might alleviate the issue. Still, an option could be welcome. I would recommend making the default the amount of logical cores, however, and not just 4. I don't think that will be an issue, really. At any time, there are dozens of processes running on your pc. Obviously you don't have that many logical cores available. Your computer is juggling threads between cores all the time, probably, and a few extra won't cause too much extra trouble, I bet! Although of course the performance of both Minecraft and EMC Shopkeeper are likely to decrease if you do this.
607 touched on this a bit already, but what you're basically suggesting here is what the operating system already does. Even if the shopkeeper was the only program running on your computer, there'll still be dozens of processes and probably hundreds of threads requesting CPU time. The OS scheduler juggles all of these requests for CPU time to try and maximise the utilisation of each core and reduce the waiting time of each task, I can't really see why there would be much benefit in a program also joining in on that. All of your logical cores will always be available for you program, it's up to the OS to decide how to use them. I think it's best just to set the number at either that number of cores, or at whatever number you really want. Otherwise you're just risking over optimising. For interest, I just checked and noticed that my browser has allocated 17 threads to just the EMC website, which is over 4 times the number of cores I have available. Obviously you understand the benefits of parallel processing, and I think because of that it's not detrimental to allocate more threads than you have cores, because if one thread finishes faster than another, then a new one can jump in rather than having that page waiting in a big queue in just one (or four or eight) threads.
Maybe I will do this: Set the number of threads to match the local computer's number of logical cores, BUT have there be 4 threads minimum, no matter how many logical cores you have. Reason being: A major benefit to having threads is that it allows EMC Shopkeeper to do other things while a thread is waiting for the rupee transaction page to be downloaded: The other threads can work on parsing the transactions out of the downloaded transaction page HTML. It can insert already-parsed transactions into the database. So, even if you have less than 4 logical cores, my guess would be that you would still benefit from having a few more threads than you have cores.
Version 0.5.25 released! Changes include: Added 1.16 items. Tweaked transaction download performance: Multiple threads are created to download and parse the rupee transaction pages from the EMC website. Before, the number of threads was hard-coded to 4. Now, it sets the thread count to match the number of logical cores on your computer (which for some people, may be higher than 4). If your computer has fewer than 4 cores, the thread count is set to 4 as before. Added the ability to define your own download thread count from the Settings menu (not necessary for most people, for power users only). Download it now!
