Fair enough. What you're trying to do may best be done off the db server. I only point it out because we had a similar problem about a year ago in a previous job where another team was trying to do a lot of data manipulation in C#: pulling the data out of postgres, performing analysis, and then pushing the results back to postgres. It was taking 25+ hours to handle 24 hours of data, and the team working on it just couldn't optimise it sufficiently. When I and my teammate were tapped to look at the problem, the first thing we each said was "stored procedure". Once we had written that, it dropped to about 1 hour to handle 24 hours of data, so I thought I should at least propose it on this thread.
It used less CPU and memory on the DB server, too, because it didn't have to serialise all that data. You might be surprised at how much less load it actually takes on the server. If you were to make your API calls, throw them into a temp table, and then use a stored procedure to injest it to the correct tables, it may actually do better than you expect. Or maybe not - there's definitely not enough information here to tell, but sometimes it takes a total algorithmic change to effect the performance gains you need, when simple tweaks are insufficient.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|