Login to Java site with WWW::Scripter charset issue

[mikedeklerk]

I'm trying to login to a java website using WWW::Scripter module. Using the WWW::Mechanize module works fine, but once authenticated the site has JavaScript functions that I need to call - hence needing Scripter.
After doing a large amount of troubleshooting I have found that the difference in packets sent from the click() event are the Content-Type varaibale.
Mechanize has: Content-Type: application/x-www-form-urlencoded\r\n
but Scripter has: Content-Type: application/x-www-form-urlencoded; charset="ISO-8859-1"\r\n
(This is viewed using WireShark)
This explains the error I am receiving from the website which is:

> error message: "ISO-8859-1" error stack trace:
> java.nio.charset.IllegalCharsetNameException: "ISO-8859-1"
>         at java.nio.charset.Charset.checkName(Charset.java:305)
>         at java.nio.charset.Charset.lookup(Charset.java:439)
>         at java.nio.charset.Charset.forName(Charset.java(Inlined Com
+piled Code))
> 
>         at java.lang.StringCoding$DecoderCache.makeDecoder(StringCod
+ing.java(Com
> piled Code))
>         at java.lang.StringCoding$1.run(StringCoding.java(Compiled C
+ode))
>         at java.security.AccessController.doPrivileged1(Native Metho
+d)
>         at java.security.AccessController.doPrivileged(AccessControl
+ler.java(Com
> piled Code))
>         at java.lang.StringCoding$DecoderCache.getDecoder(StringCodi
+ng.java(Comp
> iled Code))
>         at java.lang.StringCoding.getDecoder(StringCoding.java(Inlin
+ed Compiled Code))
>         at java.lang.StringCoding.decode(StringCoding.java(Compiled 
+Code))
>         at java.lang.String.<init>(String.java(Compiled Code))
>         at java.lang.String.<init>(String.java(Inlined Compiled Code
+))
>         at com.ibm.ws.webcontainer.servlet.RequestUtils.parsePostDat
+a(RequestUti
> ls.java(Compiled Code))
>         at com.ibm.ws.webcontainer.srt.SRTServletRequest.parseParame
+ters(SRTServ
> letRequest.java(Compiled Code))
>         at com.ibm.ws.webcontainer.srt.SRTServletRequest.getParamete
+rNames(SRTSe
> rvletRequest.java(Compiled Code))
>         at javax.servlet.ServletRequestWrapper.getParameterNames(Ser
+vletRequestW
> rapper.java(Compiled Code))
>         at org.apache.struts.util.RequestUtils.populate(RequestUtils
+.java(Compil
> ed Code))
>         at org.apache.struts.action.RequestProcessor.processPopulate
+(RequestProc
> essor.java(Compiled Code))
>         at org.apache.struts.action.RequestProcessor.process(Request
+Processor.ja
> va(Compiled Code))
>         at org.apache.struts.action.ActionServlet.process(ActionServ
+let.java(Inl
> ined Compiled Code))
>         at org.apache.struts.action.ActionServlet.doPost(ActionServl
+et.java(Comp
> iled Code))
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java(C
+ompiled Code
> ))
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java(C
+ompiled Code
> ))
>         at com.ibm.ws.webcontainer.servlet.StrictServletInstance.doS
+ervice(Stric
> tServletInstance.java(Compiled Code))
>         at com.ibm.ws.webcontainer.servlet.StrictLifecycleServlet._s
+ervice(Stric
> tLifecycleServlet.java(Compiled Code))
>         at com.ibm.ws.webcontainer.servlet.IdleServletState.service(
+StrictLifecy
> cleServlet.java(Compiled Code))
>         at com.ibm.ws.webcontainer.servlet.StrictLifecycleServlet.se
+rvice(Strict
> LifecycleServlet.java(Inlined Compiled Code))
>         at com.ibm.ws.webcontainer.servlet.ServletInstance.service(S
+ervletInstan
> ce.java(Compiled Code))
>         at com.ibm.ws.webcontainer.servlet.ValidServletReferenceStat
+e.dispatch(V
> alidServletReferenceState.java(Compiled Code))
>         at com.ibm.ws.webcontainer.servlet.ServletInstanceReference.
+dispatch(Ser
> vletInstanceReference.java(Inlined Compiled Code))
>         at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter
+(WebAppFilte
> rChain.java(Compiled Code))
>         at com.infostream.web.filters.HibernateSessionFilter.doFilte
+r(HibernateS
> essionFilter.java(Compiled Code))
>         at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFi
+lter(FilterI
> nstanceWrapper.java(Compiled Code))
>         at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter
+(WebAppFilte
> rChain.java(Compiled Code))
>         at com.infostream.web.filters.IdentificationFilter.doFilter(
+Identificati
> onFilter.java(Compiled Code))
>         at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFi
+lter(FilterI
> nstanceWrapper.java(Compiled Code))
>         at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter
+(WebAppFilte
> rChain.java(Compiled Code))
>         at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.ha
+ndleWebAppDi
> spatch(WebAppRequestDispatcher.java(Compiled Code))
>         at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.di
+spatch(WebAp
> pRequestDispatcher.java(Compiled Code))
>         at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.fo
+rward(WebApp
> RequestDispatcher.java(Compiled Code))
>         at com.ibm.ws.webcontainer.srt.WebAppInvoker.doForward(WebAp
+pInvoker.jav
> a(Compiled Code))
>         at com.ibm.ws.webcontainer.srt.WebAppInvoker.handleInvocatio
+nHook(WebApp
> Invoker.java(Compiled Code))
>         at com.ibm.ws.webcontainer.cache.invocation.CachedInvocation
+.handleInvoc
> ation(CachedInvocation.java(Compiled Code))
>         at com.ibm.ws.webcontainer.cache.invocation.CacheableInvocat
+ionContext.i
> nvoke(CacheableInvocationContext.java(Compiled Code))
>         at com.ibm.ws.webcontainer.srp.ServletRequestProcessor.dispa
+tchByURI(Ser
> vletRequestProcessor.java(Compiled Code))
>         at com.ibm.ws.webcontainer.oselistener.OSEListenerDispatcher
+.service(OSE
> Listener.java(Compiled Code))
>         at com.ibm.ws.webcontainer.http.HttpConnection.handleRequest
+(HttpConnect
> ion.java(Compiled Code))
>         at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpC
+onnection.ja
> va(Compiled Code))
>         at com.ibm.ws.http.HttpConnection.run(HttpConnection.java(Co
+mpiled Code)
> )
>         at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Com
+piled Code))

The code I'm using to test the login is:

    #!/usr/bin/perl -w
      
    use WWW::Scripter;
    
    my $urlLogin = "XXX";
        
    my $viper = WWW::Scripter->new();
            
    $viper->use_plugin('JavaScript');
    
    $viper->get($urlLogin);
          
    $viper->form_name('loginForm');
          
    $viper->field(username => "xxx");
    $viper->field(password => "xxx");
    
    $viper->click();

As mentioned before this works fine if I use WWW::Mechanize instead of WWW::Scripter. Can anyone suggest what I can do to remove the charset field in the Content-Type of the packet being POST'ed.

[Anonymous Monk]

http://search.cpan.org/grep?cpanid=SPROUT&release=WWW-Scripter-0.029&string=charset&i=1&n=1&C=3 and http://search.cpan.org/grep?cpanid=SPROUT&release=WWW-Scripter-0.029&string=Content.Type&i=1&n=1&C=3

suggest the problem is with the website, either the html or the javascript is setting the charset erroneously, and scripter obliges

So, since Scripter is a LWP subclass, you might see http://search.cpan.org/perldoc/LWP::UserAgent#Handlers and http://search.cpan.org/perldoc/HTTP::Config#Matching like

my $umf = URI->new( $urlLogin );
$scripter->add_handler(
  request_prepare => sub {
  my($request, $ua, $h) = @_
  ->content_type_charset
  return;
},
  m_scheme => $umf->scheme, # only call this handler 
  m_host => $umf->host,     # for this host ...
);

[Anonymous Monk]

Whoops, unfinished though there :)

my($type, $charset) = $h->content_type_charset;
$h->content_type( $type ); # ditch charset