Network outages and Master to Master

travistravis Junior MemberPosts: 180
I have the pleasure of maintaining a 7 master system on a Cisco Meraki network that is administered by a different contractor. Apparently there are scheduled, automatic, not-cancellable firmware updates that happen to the switches, routers, and firewalls.
When this happens, the AMX system breaks. Some of the masters won't even ping until they are rebooted. It seems like others just aren't able to re-get the M2M going again until they are rebooted.

Anyone seen anything like this?

I'm using the Cascading topology for the master to master. Route mode direct.

Comments

  • a_riot42a_riot42 AMX Wizard Posts: 1,619
    travis wrote: »
    I have the pleasure of maintaining a 7 master system on a Cisco Meraki network that is administered by a different contractor. Apparently there are scheduled, automatic, not-cancellable firmware updates that happen to the switches, routers, and firewalls.
    When this happens, the AMX system breaks. Some of the masters won't even ping until they are rebooted. It seems like others just aren't able to re-get the M2M going again until they are rebooted.

    Anyone seen anything like this?

    I'm using the Cascading topology for the master to master. Route mode direct.

    What do you mean by cascading topology? Typically if a master loses contact with a master in its URL it will keep trying indefinitely, and then reconnect once its back on the network. This sounds like a network issue more than an AMX master issue. especially if you can't ping a master.
    Paul
  • DHawthorneDHawthorne Junior Member Posts: 4,584
    I have many times seen cases where rebooting a network device breaks communications with a NetLinx master, and the only fix I have seen is rebooting the master as well. My guess is something is storing the routing path and the masters are trying to use that path instead of whatever it changed to after the network devices rebooted. But I don't know enough about what goes on under the hood to determine where or how this failure occurs. But I haven't had that problem in a very long time ... so the first thing to check is your NetLinx firmwares. Make sure they are current. If they are, the only other solution I can think of is to put a timeline in your OFFLINE event for one of the *remote* devices on the master responsible for making the connection. Start it up when the device goes off line, and if it doesn't dome back in 5 minutes, reboot the master. You may have to do it on all the masters, but I would start with the one that is responsible for making the connection.

    This is a kludgy and inelegant fix, but it will force the network data to be refreshed. I had at least one case where if I rebooted (ie., power cycled) a dumb network switch that was intermediate, that also restored my connection. Try to find the minimum amount of things to kick, then kick them when you need to.
  • cmasoncmason Junior Member Posts: 123
    If all else fails, you could always install one of these at the affected masters: http://www.digital-loggers.com/lpc.html

    It allows you to auto ping addresses, to reboot the various outlets, if a subsequent number of pings fail.

    It may not be the most ideal solution, but it's a $129 option.
  • annuelloannuello Junior Member Posts: 294
    Gday Travis,

    I've seen AMX masters and touch panels drop their IP address when our network engineers push out config changes to our Cisco switches. During the update process the Cisco gear drops all layers except hardware - i.e. the link-layer disappears, but the electrical connection is maintained. (Using OSI terminology, the physical layer remains but the data link layer disappears.) This confuses the AMX gear in a random* manner, where some units end up with an "address" of 0.0.0.0. Unplugging the network cable for at least 5 seconds and replugging it resolves the issue. (Removal for 3 seconds is not long enough!) Getting the network engineers to temporarily shut down the network port ("shut" then "no shut" in Cisco-speak) for 10 seconds also works. I try to get them to do this when they are making major updates to many rooms/buildings.

    *Whilst I keep all my firmware versions the same (per model) and try to minimize variations in AMX hardware models, the distribution of the issue is still random across my fleet. The NDX-CV7 panels seem to be more susceptible than other AMX devices.

    I doubt that M2M has any bearing on the issue you are having. It is probably a lower network-related issue and how each piece of AMX hardware copes with the network outages.

    Expanding on Dave H's thought by a miniscule amount: If you want to avoid having code to auto-reboot your master, maybe you could put a small unmanaged Ethernet switch inline to each master and cycle the power on the switch when required. One switch per master. This would break/make your network connection when required.... Of course, this assumes that 1) you have a spare relay and 2) a disconnect/reconnect of the network lead resolves the issue.

    Roger McLean
    Swinburne University.
  • pdabrowskipdabrowski Aussie Guy Posts: 184
    I can also confirm what Roger and Dave are saying, we also use Cisco switches extensively and I also see the same symptoms if the racks lose power and are then re-powered at the same time as the switches. The masters complete their boot cycle before the Cisco switches have fully booted up - probably confirming the theory from Roger about layer 1 being up early in the Cisco boot process but as there is no Layer 2 available, the master's NIC will not work until layer 1 disappears fully for a period of time (unplugging or issuing the SHUT command to the Cisco interface) and then is re-established.
  • travistravis Junior Member Posts: 180
    Interesting. This site is all G5 touchpanels and I never notice them having trouble getting back online. Only the NI series processors have trouble.

    I wonder if the new nx masters have NICs like G5 panels...
  • travistravis Junior Member Posts: 180
    Can anyone think of a good way to detect that you have no network connection on you master?

    You could telnet into 127.0.0.1 and then try to ping something.
    Or monitor your URL list and try to decide based on how many M2M connections are missing.

    I've been browsing through the netlinx help and none of the built-in commands are jumping out at me.

    GET_IP_ADDRESS always returns an address in my tests.
  • viningvining X Member Posts: 4,341
    travis wrote: »
    Can anyone think of a good way to detect that you have no network connection on you master?

    You could telnet into 127.0.0.1 and then try to ping something.
    Or monitor your URL list and try to decide based on how many M2M connections are missing.

    I've been browsing through the netlinx help and none of the built-in commands are jumping out at me.

    GET_IP_ADDRESS always returns an address in my tests.
    I have a module that pings a WAN address every 20 minutes, could also ping a LAN address. It also gets the public IP every hour but that was using the GET/ AMX.asp which Harmon has broken as far as I've seen but I could also get the IP from my own server since my system connect to the office daily to dump their daily SITREP message. Since I don't actually use the public IP returned I just commented out that portion, I use dyndns accounts for that anyway. The Ping_Get_URL or what ever it was called is on the modpedia forum although I've updated my running modules a few times since. If you want it just let me know........
  • travistravis Junior Member Posts: 180
    thank you. I'll have a look at that.

    A way to catch M2M errors like

    Line 2741 (14:40:38.422):: Closing connection to 10.10.0.33 due to duplicate (0:1:3 already exists)

    in the program would be nice too.
  • viningvining X Member Posts: 4,341
    travis wrote: »
    thank you. I'll have a look at that.

    A way to catch M2M errors like

    Line 2741 (14:40:38.422):: Closing connection to 10.10.0.33 due to duplicate (0:1:3 already exists)

    in the program would be nice too.

    Here's the current version but with out the part that get's the public ip commented out. The code posted in the modpedia section should have the missing constants, the ones that are global to all my files, if not let me know. I have another function in my code, another axi that does all my timing, dates and stuff like that that may be in the posted code but basically in there there's a function that calls every minute routines that triggers this code by calling the check minute function. You can also use send commands to call it form where ever to want and it can return a response. It should be easy to figure out.
    PROGRAM_NAME='VAV_Ping_URL_n_GET_IP'
    (***********************************************************)
    (*  FILE_LAST_MODIFIED_ON: 06/16/2015  AT: 19:16:46        *)
    (***********************************************************)
    
    #IF_NOT_DEFINED VAV_SYSTEM_CONSTANTS
    #INCLUDE 'VAV_SYSTEM_CONSTANTS.axi' 
    #DEFINE VAV_SYSTEM_CONSTANTS
    #END_IF
    
    DEFINE_DEVICE
    
    //dvPingURL_GetIP            = 0:12:0;
    //vPingURL_GetIP        = 33014:1:0;
    //vNull                  = 32768:01:0; //VIRTUAL NULL
    
    DEFINE_CONSTANT //GENERAL CONSTANTS
    
    //IP_TCP            = 1;
    //STR_CRLF[]            = {$0D,$0A};
    TELNET_TL_QUEUE                 = 21; //CHANGE TO ONE WHEN MODULE MADE
    CHAR TELNET_QUEUE_DELIM[]      = ';' 
    TELNET_URL_NAME_LEN               = 60;
    
    //from vav constants
    //IP_CLIENT_DISABLED        = 0;
    //IP_CLIENT_DISCO        = 1;
    //IP_CLIENT_PENDING        = 2;
    //IP_CLIENT_CONNECTED        = 3;
    //IP_CLIENT_DEVICE_READY    = 4;
    //IP_CLIENT_RX_DATA        = 5;
    //IP_CLIENT_TX_DATA        = 6;
    
    TELNET_PORT            = 23;
    TELNET_HTTP_PORT        = 80;
    
    CHAR TELNET_LOCALHOST[]        = '127.0.0.1';
    CHAR GET_PUBLIC_IP_URL[]    = 'amx.com';
    CHAR GET_PUBLIC_IP_PATH[]    = 'ip.asp';
    
    CHAR PINGQ_QSIZE        = 8;
    
    DEFINE_TYPE     //QUEUE
    
    STRUCTURE _sPingQ //match the VAV_SitRep_ModClient structure
         
         {
         CHAR cIP[128];  //could be domain name
         CHAR cResult[15];
         CHAR cQTime[8];
         CHAR cPTime[8];
         CHAR cRTime[8];
         CHAR cLDate[10];
         CHAR nNotify;
         DEV  dvNotify;
         }
         
    DEFINE_TYPE     //STATUS
    
    STRUCTURE _sPing //match the VAV_SitRep_ModClient structure
         
         {
         INTEGER nConState;
         INTEGER nGet_IP;
         INTEGER nWAN_OK;
         INTEGER nQHead;
         INTEGER nQTail;
         INTEGER nLstTail;
         INTEGER nQHasItem ;//not really used
         INTEGER nTLActive;
         CHAR cPublicIP[15];
         CHAR cLstPublicIP[15];
         _sPingQ sQ[PINGQ_QSIZE];
         }
    
    DEFINE_VARIABLE //STRUCTURE
    
    VOLATILE _sPing  sPing;
    
    DEFINE_VARIABLE //TELNET IP NEEDS CHANGING PER JOB <--------------------------NEEDS CHANGING PER JOB
    
    //VOLATILE LONG nTelnetTL_ProcQueue[]     = {250};
    
    VOLATILE CHAR cTelnet_IP[]         = {''}; //USE DEFAULT 127.0.0.1 //'192.168.9.150'
    // IP address of master for telnet.  If using an alternate telnet port (not 23) such as 1023 append :1023 to the IP address.
    // ex. VOLATILE CHAR cTelnet_IP[]     = {'192.168.1.60:1023'};
    // this obviously needs to correspond to the telnet port number assigned in the master.
    // or leave it blank and it will automatically use local host {'127.0.0.1'} and the default telnet port of 23.
    
    DEFINE_VARIABLE //PING ARRAY {{'yahoo.com'},{'google.com'},{'amx.com'}} CHANGE IF YOU WANT
    
    //if 1st URL fails try these next         *  1st try  *   2nd try   *   3rd try
    VOLATILE CHAR cPingURL_Arry[][30]       = {{'8.8.8.8'},{'8.8.4.4'},{'8.8.2.2'}};
    //VOLATILE CHAR cGetPublicIP_URL[]    = {'amx.com'};
    //VOLATILE CHAR cGetPublicIP_URL_[]    = {'http://amx.com/ip.asp'};
    //VOLATILE CHAR cLocalHost[]         = {'127.0.0.1'};    
    
    DEFINE_VARIABLE //TELNET USER & PASS <----------------------------------------NEEDS CHANGING PER JOB
    
    VOLATILE CHAR cTelnet_Login[][30]     = {{'USER'}, {'PASSWORD'}};    
    // login if Telnet security is active on the master, ignored if not
    // if no security leave blank or as is since it won't get used
    // ex. VOLATILE CHAR cTelnet_Login[][30] = {{''}, {''}};
    
    DEFINE_VARIABLE //GENERAL VARS
    
    //PERSISTENT INTEGER nInternet_Alive;
    VOLATILE INTEGER nPingURL_DeBug;
    VOLATILE INTEGER nTelnet_Port = 23;
    //VOLATILE INTEGER sPing.nConState;
    //VOLATILE INTEGER nTelnet_Online;
    VOLATILE INTEGER nTest_GetIP;
    //VOLATILE INTEGER sPing.nGet_IP = 0;
    
    //PERSISTENT CHAR sPing.cPublicIP[15];
    VOLATILE CHAR cTelnet_IPaddress[15];
    //VOLATILE CHAR cTelnet_TxQueue[1024];
    
    DEFINE_VARIABLE //TIME VARS
    
    //PERSISTENT SINTEGER nMY_Hour;
    //VOLATILE SINTEGER   nMY_Second;
    
    DEFINE_FUNCTION fnPingURL_DeBug(CHAR iStr[],INTEGER iAlways)
    
         {
         if(nPingURL_DeBug || iAlways)
          {
          STACK_VAR CHAR cCopyStr[2048];
          STACK_VAR INTEGER nLineCount;
          
          cCopyStr = iStr;
          
          nLineCount ++;
          WHILE(length_string(cCopyStr) > 100)
               {
               if(iAlways)
                {
                SEND_STRING 0,"'Ping_URL.axi (',itoa(nLineCount),'), ',get_buffer_string(cCopyStr,100)";
                }
               else
                {
                SEND_STRING 0,"'Ping_URL.axi (',itoa(nLineCount),') DEBUG: ',get_buffer_string(cCopyStr,100)";
                }
               nLineCount ++
               }
          if(length_string(cCopyStr))
               {
               if(iAlways)
                {
                SEND_STRING 0,"'Ping_URL.axi (',itoa(nLineCount),'), ',cCopyStr";
                }
               else
                {
                SEND_STRING 0,"'Ping_URL.axi (',itoa(nLineCount),') DEBUG: ',cCopyStr";
                }
               }
          }
         
         RETURN;
         } 
         
    DEFINE_FUNCTION fnPingURL_CheckMinute(INTEGER iMinute)//called from fnMain_EveryMinute()
    
         {
         LOCAL_VAR INTEGER nTelnet_Minute;
         
         if(nTelnet_Minute != iMinute)
          {
          nTelnet_Minute = iMinute;
          if(nTelnet_Minute%20 == 0)//0,20,40
               {
               if(sPing.nConState != IP_CLIENT_DISCO)
                {
                if(sPing.nConState > IP_CLIENT_PENDING)
                 {
                 sPing.nConState = IP_CLIENT_DISCO;
                 ip_client_close(dvPingURL_GetIP.Port);
                 wait 20
                      {
                      fnPING_Q_Job(cPingURL_Arry[1],FALSE,vNull);
                      }
                 }
                else
                 {
                 sPing.nConState = IP_CLIENT_DISCO;
                 fnPING_Q_Job(cPingURL_Arry[1],FALSE,vNull);
                 }
                }
               else
                {
                fnPING_Q_Job(cPingURL_Arry[1],FALSE,vNull);
                }
               }
          else if(nTelnet_Minute == 50)//(nTelnet_Minute%5 == 2)//
               {
               if(sPing.nConState != IP_CLIENT_DISCO)
                {
                if(sPing.nConState > IP_CLIENT_PENDING)
                 {
                 sPing.nConState = IP_CLIENT_DISCO;
                 ip_client_close(dvPingURL_GetIP.Port);
                 wait 20
                      {
                      fnConnectGET_IP();
                      }
                 }
                else
                 {
                 sPing.nConState = IP_CLIENT_DISCO;
                 fnConnectGET_IP();
                 }
                }
               else
                {
                fnConnectGET_IP();
                }
               }
          }
         if(nTest_GetIP == 1)
          {
          if(sPing.nConState != IP_CLIENT_DISCO)
               {
               if(sPing.nConState > IP_CLIENT_PENDING)
                {
                sPing.nConState = IP_CLIENT_DISCO;
                ip_client_close(dvPingURL_GetIP.Port);
                wait 20
                 {
                 fnConnectGET_IP();
                 }
                }
               else
                {
                sPing.nConState = IP_CLIENT_DISCO;
                fnConnectGET_IP();
                }
               }
          else
               {
               fnConnectGET_IP();
               }
          nTest_GetIP = 0;
          }
         else if(nTest_GetIP == 2)
          {
          if(sPing.nConState != IP_CLIENT_DISCO)
               {
               if(sPing.nConState > IP_CLIENT_PENDING)
                {
                sPing.nConState = IP_CLIENT_DISCO;
                ip_client_close(dvPingURL_GetIP.Port);
                wait 20
                 {
                 fnPING_Q_Job(cPingURL_Arry[1],FALSE,vNull);
                 }
                }
               else
                {
                sPing.nConState = IP_CLIENT_DISCO;
                fnPING_Q_Job(cPingURL_Arry[1],FALSE,vNull);
                }
               }
          else
               {
               fnPING_Q_Job(cPingURL_Arry[1],FALSE,vNull);
               }
          nTest_GetIP = 0;
          }
         } 
    
    DEFINE_FUNCTION fnPING_Q_Job(CHAR iIP[],CHAR iNotify,DEV dvNotify)
         
         {
         //fnPingURL_DeBug("'Ping Queue, Queing Index-',itoa(sPing.nQHead),', IP=',iIP,' :DEBUG<',ITOA(__LINE__),'>'",FALSE);
         STACK_VAR INTEGER nFBS;
         
         sPing.sQ[sPing.nQHead].cIP          = iIP;
         sPing.sQ[sPing.nQHead].cQTime    = TIME;
         sPing.sQ[sPing.nQHead].cLDate    = LDATE;  
         sPing.sQ[sPing.nQHead].nNotify   = iNotify;
         sPing.sQ[sPing.nQHead].dvNotify  = dvNotify;
              
         SELECT
          {
          ACTIVE(sPing.nQHead == PINGQ_QSIZE): 
               {// If head is at end
               if(sPing.nQTail <> 1) 
                {// Don't overwrite if full
                sPing.nQHead = 1;
                }
               }
          ACTIVE(sPing.nQTail <> sPing.nQHead + 1): 
               {// Don't overwrite if full
               sPing.nQHead++;
               }
          }
         
         sPing.nQHasItem = 1;
         if(sPing.nConState == IP_CLIENT_DISCO)
          {
          fnConnectTelnet();
          }
         
        // if(!TIMELINE_ACTIVE(TL_SITREP))
    //      {
    //      TIMELINE_CREATE(TL_SITREP,nSitrep_TL_Time,1,TIMELINE_ABSOLUTE,TIMELINE_REPEAT);
    //      sPing.nTLActive = 1;
    //      }
              
         RETURN;
         }
         
    DEFINE_FUNCTION fnPING_Notify(INTEGER iQIndx)
    
         {
         if(sPing.sQ[iQIndx].nNotify)//these come from elsewhere in code so broadcast the IP & result.  
          {
          SEND_STRING 0,"'Ping_URL.axi: Ping ',sPing.sQ[iQIndx].cIP,' RESULT: "',sPing.sQ[iQIndx].cResult,'" <',ITOA(__LINE__),'>'";
          if(sPing.sQ[iQIndx].dvNotify != vNull)
               {
               SEND_STRING sPing.sQ[iQIndx].dvNotify,"'PING_IP:',sPing.sQ[iQIndx].cIP,'RESULT:',sPing.sQ[iQIndx].cResult";
               }
          else
               {
               SEND_STRING vPingURL_GetIP,"'PING_IP:',sPing.sQ[iQIndx].cIP,'RESULT:',sPing.sQ[iQIndx].cResult";
               }
          }
         else//for now these are only locally driven periodic pings to test the WAN using known good IPs, yahoo, etc.
          {
          if(sPing.sQ[iQIndx].cResult == 'PASSED')
               {
               sPing.nWAN_OK = TRUE;
               }
          else
               {
               sPing.nWAN_OK = FALSE;
               }
          SEND_STRING 0,"'[ ',TIME,' ] PING, PING : ',sPing.sQ[iQIndx].cIP,' : ',sPing.sQ[iQIndx].cResult,' <',ITOA(__LINE__),'>'";
          }
         }
    
    DEFINE_FUNCTION fnPING_Ping()
    
         {
         if(sPing.nQTail <> sPing.nQHead)
          {
          fnPingURL_DeBug("'Send Ping-[ ',sPing.sQ[sPing.nQTail].cIP,' ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
          SEND_STRING dvPingURL_GetIP, "'ping ',sPing.sQ[sPing.nQTail].cIP,STR_CRLF";
          sPing.sQ[sPing.nQTail].cPTime = TIME;
          sPing.nLstTail = sPing.nQTail;
          if(sPing.nQTail < PINGQ_QSIZE)
               {
               sPing.nQTail++;
               }
          else
               {
               sPing.nQTail = 1;
               }
          }
         else
          {
          sPing.nQHasItem = 0;
          fnPingURL_DeBug("'QHasItems=0 [ send exit ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
          SEND_STRING dvPingURL_GetIP, "'exit',STR_CRLF";
          WAIT 30 'TELNET_CLOSE_CON'
               {
               if(sPing.nConState > IP_CLIENT_PENDING)
                {
                ip_client_close(dvPingURL_GetIP.Port);
                }
               //else
    //            {
    //            sPing.nConState = IP_CLIENT_DISCO;
    //            }
               }
          }      
         }
     
    DEFINE_FUNCTION fnPingURL_StartTimeLine()//not used
    
         {
        // if(!TIMELINE_ACTIVE(TELNET_TL_QUEUE) && length_string(cTelnet_TxQueue))
    //      {
    //      TIMELINE_CREATE(TELNET_TL_QUEUE,nTelnetTL_ProcQueue,1,timeline_absolute,timeline_repeat);
    //      fnPingURL_DeBug("'TIMELINE-',itoa(TELNET_TL_QUEUE),' CREATED. :DEBUG<',ITOA(__LINE__),'>'",FALSE);
    //      }
    //       
    //     RETURN;
         }
    
    DEFINE_FUNCTION fnParseTelnet(CHAR iStr[]) 
         
         {
         LOCAL_VAR INTEGER nDNR;
         LOCAL_VAR CHAR nLoginAttemp;
          
         if(sPing.nConState == IP_CLIENT_CONNECTED)
          {
          SELECT
               {
               ACTIVE(find_string(iStr, "'not authorized'", 1)):
                {
                sPing.nConState = IP_CLIENT_DISABLED;
                fnPingURL_DeBug("'Bad login *not authorized*, aborting connection. <',ITOA(__LINE__),'>'",TRUE);
                }
               ACTIVE(find_string(iStr, "'Login :'", 1)):
                {
                if(nLoginAttemp >= 5)//pretty useless if since the disable is cleared in check minute
                 {
                 fnPingURL_DeBug("'Bad login. Tries ',itoa(nLoginAttemp),', aborting connection. <',ITOA(__LINE__),'>'",TRUE);
                 sPing.nConState = IP_CLIENT_DISABLED;
                 nLoginAttemp = 0;
                 }
                else
                 {
                 fnPingURL_DeBug("'sending login name, :DEBUG<',ITOA(__LINE__),'>'",FALSE);
                 SEND_STRING dvPingURL_GetIP, "cTelnet_Login[1],STR_CRLF";
                 nLoginAttemp ++;
                 }
                }
               ACTIVE(find_string(iStr, "'Password :'", 1)):
                {
                fnPingURL_DeBug("'sending login password, :DEBUG<',ITOA(__LINE__),'>'",FALSE);
                REMOVE_STRING(iStr, "'Password :'", 1);
                SEND_STRING dvPingURL_GetIP, "cTelnet_Login[2],STR_CRLF";
                }
               ACTIVE(find_string(iStr, "'Welcome'", 1)):
                {
                nDNR = 1;
                sPing.nConState = IP_CLIENT_DEVICE_READY;
                fnPING_Ping();
                nLoginAttemp = 0;
                }
               ACTIVE(1):
                {
                fnPingURL_DeBug("'Unexpected response [ ',iStr,' ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
                }
               }
          }
         else if(sPing.nConState == IP_CLIENT_DEVICE_READY)
          {
          SELECT
               {
               ACTIVE(find_string(iStr, "'is alive'", 1)):
                {
                sPing.sQ[sPing.nLstTail].cRTime = TIME;
                sPing.sQ[sPing.nLstTail].cResult = 'PASSED';
                fnPING_Notify(sPing.nLstTail);
                if(!sPing.sQ[sPing.nLstTail].nNotify)//this was a WAN check
                 {
                 nDNR = 1;
                 }
                fnPING_Ping();
                }
               ACTIVE(find_string(iStr, "'did not respond'", 1)):
                {
                sPing.sQ[sPing.nLstTail].cRTime = TIME;
                sPing.sQ[sPing.nLstTail].cResult = 'FAILED';
                fnPING_Notify(sPing.nLstTail);
                
                if(!sPing.sQ[sPing.nLstTail].nNotify)//this was a WAN check
                 {
                 nDNR ++;
                 if(nDNR < length_array(cPingURL_Arry))//queue next IP to check WAN, maybe last server was down?
                      {
                      fnPING_Q_Job(cPingURL_Arry[nDNR],FALSE,vNull);
                      }
                 else
                      {
                      fnPingURL_DeBug("'All Ping Attempts Failed. Exiting Telnet. <',ITOA(__LINE__),'>'",TRUE);
                      fnPingURL_DeBug("'Will try again next scheduled cylce.      <',ITOA(__LINE__),'>'",TRUE);
                      fnPingURL_DeBug("'CHECK YOUR INTERNET CONNECTIONS!!!        <',ITOA(__LINE__),'>'",TRUE);
                      }
                 }
                
                fnPING_Ping();
                }
               ACTIVE(find_string(iStr, "'exit',STR_CRLF,'Unknown command'",1)):
                {
                sPing.nConState = IP_CLIENT_DISCO //repeated in offline event
                fnPingURL_DeBug("'exit "Unknown command" received, telnet is closed!'",FALSE);
                }
               ACTIVE(find_string(iStr, "'exit',STR_CRLF",1)):
                {
                sPing.nConState = IP_CLIENT_DISCO //repeated in offline event
                fnPingURL_DeBug("'exit received, telnet is closed!'",FALSE);
                }
               }
          }
          
         RETURN;
         }
         
    DEFINE_FUNCTION fnCheck_IP_n_Port(CHAR iStringIP_Port[])
    
         {
         STACK_VAR INTEGER nFBS
        
         nFBS = FIND_STRING(iStringIP_Port, "':'", 1);
        
         if(nFBS)
          {
          cTelnet_IPaddress = LEFT_STRING(iStringIP_Port, nFBS - 1);
          nTelnet_Port = ATOI(RIGHT_STRING(iStringIP_Port, LENGTH_STRING(iStringIP_Port) - nFBS));
          }
         else if(length_string(iStringIP_Port))
          {
          cTelnet_IPaddress = iStringIP_Port;
          nTelnet_Port = 23;
          }
         else
          {
          cTelnet_IPaddress = TELNET_LOCALHOST;
          nTelnet_Port = 23;
          }
          
         RETURN;
         }
        
    DEFINE_FUNCTION fnConnectGET_IP()  //use for get ip
    
         {
         LOCAL_VAR INTEGER nGet_Retry;
         
         if(sPing.nConState == IP_CLIENT_DISCO)
          {
          STACK_VAR SLONG nErrFlag;
          
          nGet_Retry = 0;
          CANCEL_WAIT 'GET_RETRY';
          CANCEL_WAIT 'GET_IP_LOG_DELAY';
          CANCEL_WAIT 'TELNET_CLOSE_CON';
          sPing.nGet_IP = TRUE;
          sPing.nConState = IP_CLIENT_PENDING;
          fnPingURL_DeBug("'Attempting to Connect GET_IP! :DEBUG<',ITOA(__LINE__),'>'",FALSE);
          nErrFlag = ip_client_open(dvPingURL_GetIP.Port,GET_PUBLIC_IP_URL,80,IP_TCP);
               {
               if(nErrFlag)
                {
                fnPingURL_DeBug("'IP_Open [ failed, err#-',fnTELNET_IP_ERROR(nErrFlag),' ] <',ITOA(__LINE__),'>'",TRUE);
                }
               else
                {
                fnPingURL_DeBug("'IP_Open [ successfull ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
                WAIT 150 'TELNET_CONNECT_TIMEOUT' 
                 {
                 fnPingURL_DeBug("'Ping_URL.axi: GET_IP, 15s, Timed Out! <',ITOA(__LINE__),'>'",TRUE);
                 if(sPing.nConState > IP_CLIENT_DISCO)
                      {
                      if(sPing.nConState > IP_CLIENT_PENDING)
                       {
                       ip_client_close(dvPingURL_GetIP.Port);
                       }
                      else
                       {
                       sPing.nConState = IP_CLIENT_DISCO;
                       }
                      }
                 }
                }
               }
          }
         else if(sPing.nConState == IP_CLIENT_DISABLED)
          {
          fnPingURL_DeBug("'Connect GET_IP, Connection Disabled? <',ITOA(__LINE__),'>'",TRUE);
          }
         else
          {
          if(nGet_Retry < 3)
               {
               fnPingURL_DeBug("'Connect GET_IP, Already Online or Pending?, Retry in 30s. <',ITOA(__LINE__),'>'",TRUE);
               WAIT 300 'GET_RETRY'
                {
                nGet_Retry++;
                fnConnectGET_IP();
                }
               }
          else
               {
               nGet_Retry = 0;
               fnPingURL_DeBug("'Connect GET_IP, Already Online or Pending?, 3 Attempts Failed! <',ITOA(__LINE__),'>'",TRUE);
               }           
          }
          
         RETURN;
         }
         
    DEFINE_FUNCTION fnPing_Get_IP()
         
         {
         if(length_string(sPing.cPublicIP) && sPing.cLstPublicIP != sPing.cPublicIP)
          {
          sPing.cLstPublicIP = sPing.cPublicIP;
          }
         sPing.cPublicIP = '';
         
         //SEND_STRING 0,"'Ping_URL.axi: Sending GET_IP [ GET /',GET_PUBLIC_IP_PATH,' HTTP/1.1 ] <',ITOA(__LINE__),'>'";
         SEND_STRING dvPingURL_GetIP,"'GET /',GET_PUBLIC_IP_PATH,' HTTP/1.1',STR_CRLF";
         SEND_STRING dvPingURL_GetIP,"'Host: ',GET_PUBLIC_IP_URL,STR_CRLF";
         SEND_STRING dvPingURL_GetIP,"STR_CRLF";
         
         fnPingURL_DeBug("'Sent: GET /',GET_PUBLIC_IP_PATH,' HTTP/1.1. :DEBUG<',ITOA(__LINE__),'>'",FALSE);
         
         RETURN;
         }
         
    DEFINE_FUNCTION CHAR[100]fnTELNET_IP_ERROR(SLONG nIP_Error) //changed name to compile  change when module builtSLONG nIP_Error)
         
         {
         STACK_VAR CHAR cErrMsg[100];
         
         SELECT
          {
          ACTIVE(nIP_Error ==  1) : // invalid server address
               {
               cErrMsg = "'IP Error: invalid server address'";
               sPing.nConState = IP_CLIENT_DISABLED;
               }
          ACTIVE(nIP_Error ==  2) : // invalid server port
               {
               cErrMsg = "'IP Error: invalid server port'";
               sPing.nConState = IP_CLIENT_DISABLED;
               }
          ACTIVE(nIP_Error ==  3) : // invalid value for Protocol
               {
               cErrMsg = "'IP Error: invalid protocol'";
               sPing.nConState = IP_CLIENT_DISABLED;
               }
          ACTIVE(nIP_Error ==  4) : // unable to open communication port with server
               {
               cErrMsg = "'IP Error: unable to open port'";
               sPing.nConState = IP_CLIENT_DISABLED;
               }
          ACTIVE(nIP_Error ==  6) : // Connection refused
               {
               cErrMsg = "'IP Error: connection refused'";  
               sPing.nConState = IP_CLIENT_DISABLED;
               }
          ACTIVE(nIP_Error ==  7) : // Connection timed out
               {
               cErrMsg = "'IP Error: connection timed out'";
               WAIT 50 'Timeout delay'
                if(sPing.nConState <> IP_CLIENT_DISABLED)
                 sPing.nConState = IP_CLIENT_DISCO;
               }
          ACTIVE(nIP_Error ==  8) : // Unknown connection error
               {
               cErrMsg = "'IP Error: unknown connection error'";
               sPing.nConState = IP_CLIENT_DISABLED;
               }
          ACTIVE(nIP_Error ==  9) : // Already closed
               {
               cErrMsg = "'IP Error: connection already closed'";
               if(sPing.nConState <> IP_CLIENT_DISABLED)
                sPing.nConState = IP_CLIENT_DISCO;
               }
          ACTIVE(nIP_Error == 10) : // Binding error
               {
               cErrMsg = "'IP Error: binding error'";
               if(sPing.nConState <> IP_CLIENT_DISABLED)
                sPing.nConState = IP_CLIENT_DISCO;
               }
          ACTIVE(nIP_Error == 11) : // Listening error
               {
               cErrMsg = "'IP Error: invalid server address'";
               sPing.nConState = IP_CLIENT_DISABLED;
               }
          ACTIVE(nIP_Error == 14) : // Local port already used
               {
               cErrMsg = "'IP Error: port in use'";
               sPing.nConState = IP_CLIENT_DISABLED;
               }
          ACTIVE(nIP_Error == 15) : // UDP socket already listening
               {
               cErrMsg = "'IP Error: UDP socket already listening'";
               sPing.nConState = IP_CLIENT_DISABLED;
               }
          ACTIVE(nIP_Error == 16) : // Too many open sockets 
               {
               cErrMsg = "'IP Error: Too many open sockets'";
               if(sPing.nConState <> IP_CLIENT_DISABLED)
                sPing.nConState = IP_CLIENT_DISCO;
               }
          }
         
         RETURN cErrMsg;
         }
    
    DEFINE_FUNCTION fnConnectTelnet() 
         
         {
         if(sPing.nConState == IP_CLIENT_DISCO)
          {
          STACK_VAR SLONG nErrFlag;
          
          CANCEL_WAIT 'TELNET_CLOSE_CON';
          
          sPing.nGet_IP = FALSE; //redundant
          sPing.nConState = IP_CLIENT_PENDING; 
          
          fnPingURL_DeBug("'Attempting to Connect! :DEBUG<',ITOA(__LINE__),'>'",FALSE);
          nErrFlag = ip_client_open(dvPingURL_GetIP.Port,cTelnet_IPaddress,nTelnet_Port,IP_TCP);
               {
               if(nErrFlag)
                {
                fnPingURL_DeBug("'Telnet Connect, IP ERROR: ',fnTELNET_IP_ERROR(nErrFlag),'. <',ITOA(__LINE__),'>'",TRUE);
                }
               else
                {
                WAIT 150 'TELNET_CONNECT_TIMEOUT' 
                 {
                 fnPingURL_DeBug("'Telnet Connect, 15s, Timed Out! <',ITOA(__LINE__),'>'",TRUE);
                 if(sPing.nConState > IP_CLIENT_DISCO)
                      {
                      if(sPing.nConState > IP_CLIENT_PENDING)
                       {
                       ip_client_close(dvPingURL_GetIP.Port);
                       }
                      else
                       {
                       sPing.nConState = IP_CLIENT_DISCO;
                       }
                      }
                 }
                }
               }
          }
         else if(sPing.nConState == IP_CLIENT_DISABLED)
          {
          fnPingURL_DeBug("'Telnet Connect. Service Disabled?. <',ITOA(__LINE__),'>'",TRUE);
          }
         else
          {
          fnPingURL_DeBug("'Telnet Connect? Already online. <',ITOA(__LINE__),'>'",TRUE);
          }
         
         RETURN;
         }
         
    DEFINE_START    //
    
    fnCheck_IP_n_Port(cTelnet_IP);
    
    sPing.nConState = IP_CLIENT_DISABLED;
    WAIT 300
         {
         sPing.nConState = IP_CLIENT_DISCO;
         }
         
    sPing.nQHead     = 1;
    sPing.nQTail     = 1;
    sPing.nQHasItem = 0;
    
    DEFINE_EVENT    //DATA_EVENT [dvPingURL_GetIP]
    
    DATA_EVENT [dvPingURL_GetIP]
         
         {
         ONLINE :
          {
          sPing.nConState = IP_CLIENT_CONNECTED;
          CANCEL_WAIT 'TELNET_CONNECT_TIMEOUT';
          CANCEL_WAIT 'Timeout delay';
          if(sPing.nGet_IP)
               {
               //SEND_STRING 0,"'Ping_URL.axi: ONLINE [ GET_IP ] <',ITOA(__LINE__),'>'";
               fnPingURL_DeBug("'ONLINE [ GET_IP ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
               fnPing_Get_IP();
               }
          else
               {
               //SEND_STRING 0,"'Ping_URL.axi: ONLINE [ PING ] <',ITOA(__LINE__),'>'";
               fnPingURL_DeBug("'ONLINE [ PING ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
               }
          }
         OFFLINE :
          {
          if(sPing.nGet_IP)
               {
               fnPingURL_DeBug("'OFFLINE [ GET_IP ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
               WAIT 50 'GET_IP_LOG_DELAY'
                {
                if(length_string(sPing.cPublicIP))
                 {
                 fnPingURL_DeBug("'[ ',TIME,' ] PING, GET IP : PUBLIC IP-[ ',sPing.cPublicIP,' ] <',ITOA(__LINE__),'>'",TRUE);
                 }
                else
                 {
                 fnPingURL_DeBug("'[ ',TIME,' ] PING, GET IP : PUBLIC IP-[ *.ERROR.*.ERROR ] <',ITOA(__LINE__),'>'",TRUE);
                 }
                }
               sPing.nGet_IP = FALSE; //reset
               }
          else
               {
               CANCEL_WAIT'TELNET_CLOSE_CON';
               fnPingURL_DeBug("'OFFLINE [ PING ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
               }
          if(sPing.nConState > IP_CLIENT_DISCO)
               {
               sPing.nConState = IP_CLIENT_DISCO;
               fnPingURL_DeBug("'OFFLINE, STATUS DISCO :DEBUG<',ITOA(__LINE__),'>'",FALSE);
               }
          else if(sPing.nConState == IP_CLIENT_DISABLED)
               {
               fnPingURL_DeBug("'OFFLINE, STATUS DISABLED <',ITOA(__LINE__),'>'",TRUE);
               }
          } 
         ONERROR :
          {
          sPing.nGet_IP = FALSE; //reset
          CANCEL_WAIT 'TELNET_CONNECT_TIMEOUT';
          SEND_STRING 0,"'Ping_URL.axi: ',fnTELNET_IP_ERROR(TYPE_CAST(DATA.NUMBER)),' <',ITOA(__LINE__),'>'";
          }
         STRING : 
          {
          if(sPing.nGet_IP)
               {
               LOCAL_VAR CHAR cGetIP_RX[3600];//full MTU
               STACK_VAR INTEGER nFBS;
               
               cGetIP_RX = "cGetIP_RX,DATA.TEXT";
               fnPingURL_DeBug("'GET IP, RX-[ ',cGetIP_RX,' ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
               nFBS = find_string(cGetIP_RX,"STR_CRLF,STR_CRLF",1);
               if(nFBS)
                {
                REMOVE_STRING(cGetIP_RX,"STR_CRLF,STR_CRLF",1);
                sPing.cPublicIP = cGetIP_RX;
                sPing.nWAN_OK = TRUE;
                //sPing.nGet_IP = FALSE; //moved to offline //reset when ip found
                cGetIP_RX = "''";
                WAIT 30
                 {
                 if(sPing.nConState > IP_CLIENT_PENDING)
                      {
                      ip_client_close(dvPingURL_GetIP.Port);//expidite the closing.  hangs too long
                      }
                 }
                }
               else
                {
                fnPingURL_DeBug("'RX Err! RX-[ ',cGetIP_RX,' ] <',ITOA(__LINE__),'>'",TRUE);
                }
               }
          else
               {
               fnPingURL_DeBug("'RX: [ ',DATA.TEXT,' ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
               if(length_string(DATA.TEXT) > 3)//don't process telnet echoes
                {
                fnPingURL_DeBug("'RX: [ ',DATA.TEXT,' ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
                fnParseTelnet(DATA.TEXT);
                }
               }
          }
         }
    
    DEFINE_EVENT    //DATA_EVENT [vPingURL_GetIP]
    
    DATA_EVENT [vPingURL_GetIP]
         
         {
         COMMAND://command should installed in modules that will send strings to this file
          {
          //example
          //SEND_COMMAND vPingURL_GetIP,"'PING_IP:192.168.1.2'"  OR DOMAIN NAME
          //SEND_COMMAND vPingURL_GetIP,"'PING_IP:192.168.1.2NOTIFY:33021:1:0'"  OR DOMAIN NAME
          // RESPONSE:
          //'PING_IP:192.168.1.2RESULT:PASSED'
          //'PING_IP:192.168.1.2RESULT:FAILED'
          fnPingURL_DeBug("'CMD RX CMD: ',DATA.TEXT,' :DEBUG<',ITOA(__LINE__),'>'",FALSE);
          SELECT
               {
               ACTIVE(find_string(DATA.TEXT,"'PING_IP:'",1)):
                {
                STACK_VAR INTEGER nFBS;
                
                REMOVE_STRING(DATA.TEXT,"'PING_IP:'",1);
                
                nFBS = find_string(DATA.TEXT,"'NOTIFY:'",1);
                if(nFBS)
                 {
                 STACK_VAR CHAR cIP[128];//COULD BE LONG DOMAIN NAME
                 STACK_VAR INTEGER nNum;
                 STACK_VAR INTEGER nPort;
                 STACK_VAR INTEGER nSys;
                 
                 cIP = GET_BUFFER_STRING(DATA.TEXT,nFBS-1);
                 REMOVE_STRING(DATA.TEXT,"'NOTIFY:'",1);
                 
                 nNum  = atoi(REMOVE_STRING(DATA.TEXT,"':'",1));
                 nPort = atoi(REMOVE_STRING(DATA.TEXT,"':'",1));
                 nSys  = atoi(DATA.TEXT);
                 
                 fnPingURL_DeBug("'RX request to PING_IP: [ ',cIP,' ], respond to [ ',fnDev_To_String(nNum:nPort:nSys),' ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
                 fnPING_Q_Job(cIP,TRUE,nNum:nPort:nSys);
                 }
                else
                 {
                 fnPingURL_DeBug("'RX request to PING_IP: [ ',DATA.TEXT,' ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
                 fnPING_Q_Job(DATA.TEXT,TRUE,vNull);
                 }
                }
               ACTIVE(find_string(DATA.TEXT,"'GET_IP:'",1)):
                {
                fnPingURL_DeBug("'RX request [ ',DATA.TEXT,' ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
                REMOVE_STRING(DATA.TEXT,"'GET_IP:'",1);
                fnConnectGET_IP();
                }
               ACTIVE(1):
                {
                fnPingURL_DeBug("'RX unkown request: [ ',DATA.TEXT,' ] :DEBUG<',ITOA(__LINE__),'>'",FALSE);
                }
               }
          }
         STRING: 
          {
          fnPingURL_DeBug("'CMD RX STR: [ ',DATA.TEXT,' ] <',ITOA(__LINE__),'>'",TRUE);
          }
         }
    
    DEFINE_PROGRAM //EMPTY (some stuff commented out)
    
    
Sign In or Register to comment.