Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: Paging with REST::Client?

by sundialsvc4 (Abbot)
on Jan 13, 2020 at 05:14 UTC ( #11111349=note: print w/replies, xml ) Need Help??


in reply to Paging with REST::Client?

Allowed me to write a new daemon that checks for queued changes and decides whether to kick off a DHCP restart ... Huh?!

In all my many years, I have never seen fit to “restart a DHCP daemon ... anywhere on any device” in order to get my intended work done.   And so, if you think that you are routinely needing to do that, you are IMHO already fighting a losing battle against the fire.

Reading further:   “Allowed me to write a new daemon that checks for queued changes and decides whether to kick off a DHCP restart.   It usually works great, but there are times when we are importing a ton of changes (e.g. 1800+) and I'm hitting the max amount of data REST::Client will accept.”

Okay, there’s your actual problem:   throttling.   As in to say that “your present strategy, being based on simple curl and on request-loads that are routinely much smaller, was never properly engineered to intelligently handle.”   Your simplistic present strategy is what I call “flaming arrows” ... when a new request arrives, ignite another flaming arrow, launch it into the air and hope for the best.

Maybe that was, at its time, a great strategy.   Maybe it worked – maybe for years.   Not anymore.   Strategies like this one do not “scale up.”

What you actually have here – on the requestor side, not the server side – is a basic workload management problem, for which a plentitude of solutions already exist in CPAN.   No matter how many thousands of changes your system might be requested to do at any one time, your software must throttle, and therefore effectively manage, that “queue.”   It must present an appropriate volume of requests to the back-end server and be able to hold the rest of them back, so as to maintain a predictable and sustainable level of service.   Exactly as every fast-food restaurant does every single day at lunch hour.

What you need to select and deploy right now is a workload-management system which puts these 1,800+ requests into a queue which is then serviced by an algorithm that knows how to manage the request stream to the back-end side.   CPAN is full of excellent candidates.   Perhaps other Monks would like to now suggest their favorites?

Replies are listed 'Best First'.
Re^2: Paging with REST::Client?
by Argel (Prior) on Jan 13, 2020 at 20:45 UTC

    For anyone curious, we're using a third party product that uses ISC DHCP under the hood. Any change to a DHCP scope requires a DHCP restart, and we have tens of thousands of scopes as we are a large enterprise grade company. Between broke/fix issues, new branches, and hardware refreshes that require scope changes, we have a lot of changes occurring in our environment. So updating 1800 scopes via the app's CSV bulk import process is tame and fully supported.

    Reading between the lines from our vendor support, there are a couple race conditions that can occur when queuing up too many restarts. And as per above, we have a lot of changes going on daily. So after yet another disruptive outage last fall, I wrote a restart daemon that checks if a restart is needed and if so,is a restart in progress. If there is one in progress it skips restarting DHCP, if not then it restarts it. So now we rarely queue up any restarts (a queued up restart occurs when a restart is in progress and someone requests a restart).

    Sundialsvc4, good intentions don't guarantee a good answer. Instead of asking for clarifications, you made a lot of incorrect assumptions and ran with them, resulting in an irrelevant post. Hopefully given the above you can see that some curiosity could lead to asking for clarifications, which in turn could lead to a useful response.

    Elda Taluta; Sarks Sark; Ark Arks
    My deviantART gallery

      "Hopefully given the above you can see that some curiosity could lead to asking for clarifications, which in turn could lead to a useful response."

      Lower your expectations....

Re^2: Paging with REST::Client?
by 1nickt (Abbot) on Jan 13, 2020 at 11:17 UTC

    Mike, do you ever compare your posts to the others on this forum? Do you notice any consistent qualitative difference?

    It usually works great, but there are times when we are importing a ton of changes (e.g. 1800+) and I'm hitting the max amount of data REST::Client will accept.”
    Okay, there’s your actual problem: throttling.

    Wrong. Your analysis of the problem and your comprehension of the posts you are replying to is utterly wrong. The OP needs paging in handling a large HTTP response (as s/he well understood); it's nothing whatsoever to do with throttling requests to a server of any kind.

    The rest of your comments, even if taken in the context of a discussion about throttling, are ignorant, vague, narcissistic, condescending, pompous and worthless all at the same time. Given your misunderstanding of what the topic even is, they are also irrelevant.

    Suggestion: after a thought leaps into your head from 20 years ago and you craft one of your advice columns, before posting, open a separate window, re-review the thread you plan to reply to, and double-check whether you are at least speaking on the same subject.


    The way forward always starts with a minimal test.
Re^2: Paging with REST::Client?
by Anonymous Monk on Jan 13, 2020 at 10:16 UTC
    I'd suggest not taking any of the above to heart. One of your worst posts in a long time, though you managed to login this time around.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11111349]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (3)
As of 2020-02-21 10:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What numbers are you going to focus on primarily in 2020?










    Results (94 votes). Check out past polls.

    Notices?