Downolad of Complex Websites and Archives for further analysis

I've used HTTrack in the past and it works pretty well for that.

Thank you very much. I am trying this. Well it is not working with sub pages. But the program is downloading entire database (I guess), lets see what happens. Thank you again!

You may have to tweak the settings a bit to get what you want.

Have you contemplated actually contacting the UN? They might assist. Depending on your purposes.

Sulkhan - if you can give any indication of the order of magnitude involved, many posters here can answer you. Tera- , Peta-, whatever. And btw, HOW did YOU (never heard of you before, for the record) come to have a "negative" rating of MINUS 50?

Not that it matters to anyone in mathematics (we DO use Arabic numerals) but I'm just curious. Tks.

O yes, I did, actually they cannot do much about that. They have huge data base and are quite confused with the amount of work (it is my impression).

Don't worry about the ratings, some old members don't like them (they are a new feature) and are trying to stir up more angst about them by rating new people down and pointing out how it's unhelpful to the community.

This will pass as their maturity that deserted them returns.

I do not really have the idea how this site exactly works, just needed online help and used the search engine. Why people here do not like me? No idea, maybe I have too complicated name .

To be honest I cannot excatly understand what you mean with "order of magnitude", I am sorry.

Well that can be possible explanation also

The question was about mathematics, which you signally failed to grasp.

YOU may be one of those OLDER members who can't grasp a goddam THING, MR (dr?) Gentel, but your I DI O TIC comment couldn't possibly apply to METADATA - look it UP ya DIMWIT.

Sorry to the original poster - I know the UN database and will get back to you.

well Sulkhan, no offense, but if you got passwords for accessing UN databases, then surely you got access to their technical support as well

If you do NOT have access to the databases then what IS your problem? Pls start at the beginning. Thanks

If you can do this on a Linux box, you also might want to try wget. And here is an article that helps explain the options you'd use to download a whole website:

http://linuxreviews.org/quicktips/wget/

Be careful with your rules, primarily about what offsite links it will follow. When I've spidered sites I often stop the spider right away when I find it is too broad and would take too long (after all, you can end up telling it to try to spider the whole internet) or tweak the priorities and thresholds to get it to the pages I need it to go to.

I am trying this program (you kindly provided the link). Well, I do not know what is wrong. It does not seem to be so difficult that I cannot understand. Actually what I want is very simple thing: I would like to download the database in order dont to jump from page to page what costs me too much time. Funny that it is so difficult to find some tool which can perform such an easy task.

Now something is worng with the program or with the database: I want to download database by the country. Lets say USA. So I enter the link in the program and give the order to download the US database (the sites only). Now I am waiting for 45 minutes - the program has downloaded 50 MB-s but still continuing to work. 45 minutes per country and it is not done jet? then it is better to do everything manually again. Confused...

Can you give a link to what you are trying to download? Or is it behind a membership login?

No, it is free to access now. Sure here is the link:

http://157.150.195.4/LibertyIMS::/sidqLpzWYHM3DQ9iNSQ/cmd%3DXmlGetWebPage%3BCmdFile%3DXmlAdvSearch.cmd

This is advanced search mode. After you can select any country from the list and simply press seacrh button leaving everything else default. So you will get results which I would like to download. Exammple of Canada is here:

http://157.150.195.4/LibertyIMS::/sidqLpzWYHM3DQ9iNSQ/Cmd%3D%24%24B8D1gZoxtumv91dda%3BAvBV%3D%23fP

That is what I would like to download - complete results (my goal is plain text of it at the end).

Sometimes these links fail to work. In that case you can go to the page with fallowing steps. go to: http://untreaty.un.org/

Press English

Go to the end of the page and press: Access to Databases

From quick links pick up United Nations Treaty Series

Press Advanced Mode. There you are. Pick up any country you like. So how can I save that database?

Now I see where it's having problems. The spiders I gave you as recommendations don't do well with this kind of "walled garden" and spiders don't do well with data hidden behind POST forms and not using the URL to reference them in any way (e.g. your canada example doesn't show the info you saw when I visit your link).

It's a much more complicated problem to solve because it requires the spider to emulate a human a bit more. Even Google is just now starting to test this kind of crawling with their spiders.

There are some web content extractors I have heard of that may be able to help you, but they often cost a few hundred dollars and you still need to train it in ways that might be more complex than your task merits.

So my recommendation now is that you look for software that lets you make macros or automate your own actions.

Things like Auto Hot Key or for a great Mac example the Automator software for macs that allow you to use your mouse to train the computer to repeat your actions may be useful.

I don't have personal experience using any of them the exact way you are trying but would recommend you start with the Auto Hot Key (link above) and this one (in the free browser versions):

http://www.iopus.com/imacros/compare/all/

Thing is, there's really no way around needing to do some programmatic work to automate your manual work, because what you are doing isn't that easy for a computer to replicate without precise instructions and in this case a website downloader or spider isn't going to work, and you are in the realm of content scrapers and macros if you really want to automate this.