[AC-4245] Arrow application build crashes due to timeout during NPM install

GitHub Issue	n/a
Type	Bug
Priority	n/a
Status	Closed
Resolution	Fixed
Resolution Date	2016-08-31T20:19:14.000+0000
Affected Version/s	n/a
Fix Version/s	n/a
Components	Arrow Builder
Labels	n/a
Reporter	Parijat sahai
Assignee	Shak Hossain
Created	2016-07-26T15:46:57.000+0000
Updated	2016-08-31T20:19:14.000+0000

Description

I'm getting the following error frequently during the build:

2016-07-26T11:36:22-04:00 | npm ERR! Linux 3.13.0-88-generic|
2016-07-26T11:36:22-04:00 | npm ERR! node v0.12.4|
2016-07-26T11:36:22-04:00 | npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "i" "--production" "--unsafe-perm"|
2016-07-26T11:36:22-04:00 | npm ERR! npm  v2.10.1|
2016-07-26T11:36:22-04:00 | npm ERR! code ECONNRESET|
2016-07-26T11:36:22-04:00 | npm ERR! errno ECONNRESET|
2016-07-26T11:36:22-04:00 | npm ERR! syscall read|
2016-07-26T11:36:22-04:00 | npm ERR! network read ECONNRESET|
2016-07-26T11:36:22-04:00 | npm ERR! network and is related to network connectivity.|
2016-07-26T11:36:22-04:00 | npm ERR! network This is most likely not a problem with npm itself|
2016-07-26T11:36:22-04:00 | npm ERR! network |

Please advise what is going on. After several attempts the build succeeds, so clearly the issue is elsewhere.

Comments

Sharif AbuDarda 2016-07-26

Hello, Please provide details of your environment. Also, it would be helpful if you provide a sample code that regenerates the issue. Thanks.

Parijat sahai 2016-07-26

What kind of details? Would the following help?

   App name: ReconnexCloud
    -- Created by: parijat@aluminate.io
    -- URL: https://38dce6d6f0f509b6c1f6a5a3390c92e06399fdfe.cloudapp-enterprise.appcelerator.com
    -- DOMAIN: cloudapp.rekonnex.com
    -- Created at: Wed Aug 05 2015 15:17:34 GMT-0400 (EDT)
    -- Published at: 2016-07-26T10:41:33-04:00
    -- Maximum allowed number of servers: 1
    -- Minimum number of servers: 1
    -- Number of desired servers: 1
    -- Auto scale-up enabled: true
    -- Maximum queued requests per server: 50
    -- Auto scale-down enabled: true
   
   -- Active deployment: 
       - Node version: 0.12.4
       - Server size: Medium
       - App version: 1.0.14
       - Deployed at: Tue Jul 26 2016 10:43:39 GMT-0400 (EDT)
       - Status: failed
       - Server errors: 579776a175a5014bf3da7a7e: App crashed.
   Error: Error loading connector/appc.arrowdb. Request returned with HTTP status code 503 503 Service Unavailable. No server is available to handle this request.
   at /opt/app/node,modules/arrow/lib/arrow.js:516:11
   at Connector.connectCallback (/opt/app/node,modules/arrow/node,modules/arrow-orm/lib/connector.js:464:21)
   at /opt/app/node,modules/arrow/node,modules/arrow-orm/node,modules/async/lib/async.js:251:17
   at /opt/app/node,modules/arrow/node,modules/arrow-orm/node,modules/async/lib/async.js:148:21
   at /opt/app/node,modules/arrow/node,modules/arrow-orm/node,modules/async/lib/async.js:248:21
   at /opt/app/node,modules/arrow/node,modules/arrow-orm/node,modules/async/lib/async.js:612:34
   at /opt/app/node,modules/connectors/appc.arrowdb/lib/lifecycle/connect.js:16:11
   at Object.<anonymous> (/opt/app/node,modules/connectors/appc.arrowdb/node,modules/arrowdb/lib/rest.js:313:5)
   at Request.,callback (/opt/app/node,modules/connectors/appc.arrowdb/node,modules/arrowdb/lib/rest.js:166:11)
   at Request.self.callback (/opt/app/node,modules/connectors/appc.arrowdb/node,modules/arrowdb/node,modules/request/request.js:187:22)
   
   
   
                        579776f875a5014bf3da7a80: App crashed.
   TypeError: Cannot read property 'map' of undefined
   at ,map (/opt/app/node,modules/connectors/appc.composite/node,modules/async/lib/async.js:51:16)
   at ,asyncMap (/opt/app/node,modules/connectors/appc.composite/node,modules/async/lib/async.js:234:15)
   at Object.map (/opt/app/node,modules/connectors/appc.composite/node,modules/async/lib/async.js:216:23)
   at Connector.runJoin (/opt/app/node,modules/connectors/appc.composite/lib/utility/runJoin.js:22:10)
   at modelsDoneCallback (/opt/app/node,modules/connectors/appc.composite/lib/utility/execComposite.js:113:9)
   at done (/opt/app/node,modules/connectors/appc.composite/node,modules/async/lib/async.js:132:19)
   at /opt/app/node,modules/connectors/appc.composite/node,modules/async/lib/async.js:32:16
   at methodCallback (/opt/app/node,modules/connectors/appc.composite/lib/utility/execComposite.js:105:5)
   at Model.<anonymous> (/opt/app/node,modules/connectors/appc.arrowdb/lib/schema/wireModel.js:241:14)
   at Object.<anonymous> (/opt/app/node,modules/connectors/appc.arrowdb/node,modules/arrowdb/lib/rest.js:313:5)

Parijat sahai 2016-07-26

Please look at the edited description of the ticket. The error I'm now consistently getting during the deployment is that one. Now my app simply won't deploy. Nothing has changed at our end, so it's strange that this issue has come up.

Parijat sahai 2016-07-27

Do you have any status update on this issue? Any workaround or any fix? I just upgraded the Arrow CLI to version 5.3.1 hoping that might resolve the issue but no change - the deployment fails every time at the same place and I really need to get past this issue:

   2016-07-27T11:48:28-04:00 | ├── tar.gz@1.0.5 (commander@2.9.0, tar@2.2.1, bluebird@2.10.2, fstream@1.0.10, mout@0.11.1)|
   2016-07-27T11:48:28-04:00 | ├── appcelerator-saml@0.5.0 (semver@5.3.0, cookies@0.5.1, passport@0.3.2, passport-saml@0.15.0)|
   2016-07-27T11:48:28-04:00 | ├── babel@5.8.38 (slash@1.0.0, path-is-absolute@1.0.0, path-exists@1.0.0, fs-readdir-recursive@0.1.2, convert-source-map@1.3.0, commander@2.9.0, output-file-sync@1.1.2, glob@5.0.15, chokidar@1.6.0, source-map@0.5.6, babel-core@5.8.38)|
   2016-07-27T11:48:28-04:00 | ├── react@0.14.8 (envify@3.4.1, fbjs@0.6.1)|
   2016-07-27T11:48:28-04:00 | ├── acs@1.2.1 (colors@0.6.0-1, temp@0.4.0, commander@0.6.1, wrench@1.3.9, pkginfo@0.2.2, node-uuid@1.3.3, mkdirp@0.3.3, string@1.1.0, ejs@0.8.0, underscore@1.3.3, tar@0.1.13, fstream@0.1.18, log4js@0.5.1, uglify-js@1.3.2, jade@0.28.1, connect@2.3.5, request@2.14.0, express@3.0.0, esprima@0.9.9, moment@2.8.3, socket.io@0.9.16, npm@2.14.22)|
   2016-07-27T11:48:28-04:00 | └── arrow-admin@0.7.11 (mkdirp@0.5.1, spdx-licenses@0.0.3, appc-platform-sdk@1.3.6, arrow-util@0.1.6, arrow-docgen@0.3.3, lodash@4.14.0)|
   2016-07-27T11:48:28-04:00 | npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "i" "--production" "--unsafe-perm"|
   2016-07-27T11:48:28-04:00 | npm ERR! Linux 3.13.0-88-generic|
   2016-07-27T11:48:28-04:00 | npm ERR! node v0.12.4|
   2016-07-27T11:48:28-04:00 | npm ERR! npm  v2.10.1|
   2016-07-27T11:48:28-04:00 | npm ERR! If you need help, you may report this error at:|
   2016-07-27T11:48:28-04:00 | |
   2016-07-27T11:48:28-04:00 | npm ERR! |
   2016-07-27T11:48:28-04:00 | npm ERR! Callback called more than once.|
   2016-07-27T11:48:28-04:00 | npm ERR!     <https://github.com/npm/npm/issues>;|
   2016-07-27T11:48:28-04:00 | |
   2016-07-27T11:48:28-04:00 | npm install exit with code 1|
   2016-07-27T11:48:28-04:00 | {"success":false,"errcode":219,"message":"Failed to install dependencies."}

Parijat sahai 2016-07-27

I decided to upgrade the Appcelerator to 5.3.1 and also ran "npm update -g" to bring all modules up to date. Not sure if that caused it, but for some reason, my application successfully deployed this time. That said, the "acs list" and other "acs" CLI commands have stopped working with some node.js error on "form-data" module. I have started to use "appc cloud list" and similar commands as a result. Any ideas?
Parijat sahai 2016-07-28

Well, the deployment errors came back again yesterday and it wouldn't deploy for hours of trying. For some reason, just a few minutes ago, the application successfully deployed. However, the application crashes almost instantly when any API calls are made that cause it to access the ArrowDB database. So, until the ArrowDB issue is resolved, this problem isn't over it seems.
Ingo Muschenetz 2016-07-28

I believe the root cause of at least the initial report of this is CLOUDSRV-4875, as issues with ArrowDB from the queries were bringing down registry server (which would affect logging, in, downloading connectors, and deployments) That has been resolved, so I would be curious to know what the new set of issues are. Do you have an updated stack trace?

Barry Green 2016-07-29

I'm also getting the exact same "Failed to install dependencies." error as shown in the second log window of the original report above. No dependency errors are actually reported, and if I re-run the same command several times, eventually it succeeds. See log below:

   2016-07-29T17:35:31+10:00 | ├── cssom@0.3.1|
   2016-07-29T17:35:31+10:00 | ├── slick@1.12.2|
   2016-07-29T17:35:31+10:00 | └── cheerio@0.20.0 (dom-serializer@0.1.0, entities@1.1.1, css-select@1.2.0, htmlparser2@3.8.3, jsdom@7.2.2, lodash@4.14.0)|
   2016-07-29T17:35:31+10:00 | ├── cross-spawn-async@2.2.4 (lru-cache@4.0.1, which@1.2.10)|
   2016-07-29T17:35:31+10:00 | ├── web-resource-inliner@2.0.0 (xtend@4.0.1, async@0.9.2, clean-css@1.1.7, datauri@0.2.1, htmlparser2@3.9.1, uglify-js@2.7.0, cli-color@0.3.3)|
   2016-07-29T17:35:31+10:00 | ├── dev-null@0.1.1|
   2016-07-29T17:35:31+10:00 | ├── uuid-v4@0.1.0|
   2016-07-29T17:35:31+10:00 | arrow@1.8.2 node_modules/arrow|
   2016-07-29T17:35:31+10:00 | ├── pluralize@1.2.1|
   2016-07-29T17:35:31+10:00 | ├── react-dom@0.14.8|
   2016-07-29T17:35:31+10:00 | ├── xml@1.0.0|
   2016-07-29T17:35:31+10:00 | ├── cookie-parser@1.4.3 (cookie-signature@1.0.6, cookie@0.3.1)|
   2016-07-29T17:35:31+10:00 | ├── appc-marked@0.3.6|
   2016-07-29T17:35:31+10:00 | ├── mustache@2.2.1|
   2016-07-29T17:35:31+10:00 | ├── chalk@1.1.3 (escape-string-regexp@1.0.5, supports-color@2.0.0, ansi-styles@2.2.1, has-ansi@2.0.0, strip-ansi@3.0.1)|
   2016-07-29T17:35:31+10:00 | ├── wrench@1.5.9|
   2016-07-29T17:35:31+10:00 | ├── json2csv@3.6.2 (path-is-absolute@1.0.0, lodash.flatten@4.3.0, lodash.uniq@4.4.0, lodash.get@4.4.0, commander@2.9.0, flat@2.0.1, cli-table@0.3.1)|
   2016-07-29T17:35:31+10:00 | ├── client-sessions@0.7.0 (cookies@0.5.0)|
   2016-07-29T17:35:31+10:00 | ├── require-new@1.1.0 (stack-trace@0.0.9)|
   2016-07-29T17:35:31+10:00 | ├── compression@1.6.2 (on-headers@1.0.1, vary@1.1.0, bytes@2.3.0, compressible@2.0.8, accepts@1.3.3)|
   2016-07-29T17:35:31+10:00 | ├── ejs@2.5.1|
   2016-07-29T17:35:31+10:00 | ├── numeral@1.5.3|
   2016-07-29T17:35:31+10:00 | ├── body-parser@1.15.2 (bytes@2.4.0, content-type@1.0.2, depd@1.1.0, http-errors@1.5.0, on-finished@2.3.0, qs@6.2.0, raw-body@2.1.7, type-is@1.6.13, iconv-lite@0.4.13)|
   2016-07-29T17:35:31+10:00 | ├── forever-monitor@1.7.0 (minimatch@2.0.10, ps-tree@0.0.3, utile@0.2.1, chokidar@1.6.0, broadway@0.3.6)|
   2016-07-29T17:35:31+10:00 | ├── js-yaml@3.6.1 (esprima@2.7.2, argparse@1.0.7)|
   2016-07-29T17:35:31+10:00 | ├── connect-busboy@0.0.2 (busboy@0.2.13)|
   2016-07-29T17:35:31+10:00 | ├── arrow-objectmodel@0.2.21 (js-beautify@1.6.3)|
   2016-07-29T17:35:31+10:00 | ├── appc-ldapauth@2.3.3 (lru-cache@2.5.0, bcryptjs@2.1.0, appc-ldapjs@0.7.3)|
   2016-07-29T17:35:31+10:00 | ├── appc-logger@1.1.27 (response-time@2.3.1, readable-stream@2.1.4, fs-extra@0.26.7, bunyan@1.8.1)|
   2016-07-29T17:35:31+10:00 | ├── highlight.js@9.5.0|
   2016-07-29T17:35:31+10:00 | ├── tar.gz@1.0.5 (commander@2.9.0, tar@2.2.1, bluebird@2.10.2, fstream@1.0.10, mout@0.11.1)|
   2016-07-29T17:35:31+10:00 | ├── appcelerator-saml@0.5.0 (semver@5.3.0, cookies@0.5.1, passport@0.3.2, passport-saml@0.15.0)|
   2016-07-29T17:35:31+10:00 | ├── arrow-orm@0.6.10 (lru-cache@2.7.3, async@0.9.2, mingo@0.3.3)|
   2016-07-29T17:35:31+10:00 | └── arrow-admin@0.7.11 (mkdirp@0.5.1, spdx-licenses@0.0.3, appc-platform-sdk@1.3.7, arrow-util@0.1.6, arrow-docgen@0.3.3, lodash@4.14.0)|
   2016-07-29T17:35:31+10:00 | ├── react@0.14.8 (envify@3.4.1, fbjs@0.6.1)|
   2016-07-29T17:35:31+10:00 | ├── babel@5.8.38 (slash@1.0.0, path-is-absolute@1.0.0, path-exists@1.0.0, fs-readdir-recursive@0.1.2, convert-source-map@1.3.0, commander@2.9.0, output-file-sync@1.1.2, glob@5.0.15, chokidar@1.6.0, source-map@0.5.6, babel-core@5.8.38)|
   2016-07-29T17:35:31+10:00 | ├── acs@1.2.1 (colors@0.6.0-1, temp@0.4.0, commander@0.6.1, wrench@1.3.9, pkginfo@0.2.2, node-uuid@1.3.3, mkdirp@0.3.3, string@1.1.0, ejs@0.8.0, underscore@1.3.3, tar@0.1.13, fstream@0.1.18, log4js@0.5.1, uglify-js@1.3.2, jade@0.28.1, connect@2.3.5, request@2.14.0, express@3.0.0, esprima@0.9.9, moment@2.8.3, socket.io@0.9.16, npm@2.14.22)|
   2016-07-29T17:35:31+10:00 | npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "i" "--production" "--unsafe-perm"|
   2016-07-29T17:35:31+10:00 | npm ERR! node v0.12.7|
   2016-07-29T17:35:31+10:00 | npm ERR! Linux 3.13.0-88-generic|
   2016-07-29T17:35:31+10:00 | npm ERR! Callback called more than once.|
   2016-07-29T17:35:31+10:00 | npm ERR! npm  v2.11.3|
   2016-07-29T17:35:31+10:00 | |
   2016-07-29T17:35:31+10:00 | npm ERR!     <https://github.com/npm/npm/issues>;|
   2016-07-29T17:35:31+10:00 | npm ERR! If you need help, you may report this error at:|
   2016-07-29T17:35:31+10:00 | npm ERR! |
   2016-07-29T17:35:31+10:00 | |
   2016-07-29T17:35:31+10:00 | npm install exit with code 1|
   2016-07-29T17:35:31+10:00 | {"success":false,"errcode":219,"message":"Failed to install dependencies."}

Ingo Muschenetz 2016-07-29

When you are "running the build" what command are you using? Can I also see a copy of your package.json of your project?
Barry Green 2016-07-29

I run "appc publish --force" to build. Contents of package.json are below. {noformat} { "name": "my_arrow", "description": "", "version": "1.0.0", "author": "BG", "license": "", "framework": "none", "keywords": [ "appcelerator", "arrow" ], "repository": {}, "private": true, "dependencies": { "arrowdb": ">=1.0.6", "async": "^1.4.0", "lodash": "^3.10.1", "pkginfo": "^0.3.0", "stripe": "^4.6.0", "validator": "^5.2.0", "twilio": "^2.9.1", "request": "^2.72.0", "sendgrid": "^3.0.7", "handlebars": "^4.0.5", "juice": "^2.0.0", "moment": "^2.13.0", "csprng": "^0.1.1" }, "devDependencies": { "grunt": "^0.4.5", "grunt-contrib-clean": "^0.6.0", "grunt-contrib-jshint": "^0.10.0", "grunt-kahvesi": "^0.1.0", "grunt-mocha-test": "^0.11.0", "mocha": "^1.21.4", "should": "^4.0.4" }, "main": "app.js", "healthCheck": true, "engines": { "node": "0.12.7" } } {noformat}

Parijat sahai 2016-07-29

I'm using the same command as what Barry mentioned. My package.json is the following:

    {
      "name": "MyArrowApp",
      "description": "",
      "version": "1.0.14",
      "author": "Parijat Sahai",
      "license": "",
      "framework": "none",
      "keywords": [
        "appcelerator",
        "arrow"
      ],
      "repository": {},
      "private": true,
      "dependencies": {
        "arrowdb": "^1.0.6",
        "async": "^0.9.0",
        "azure-sb": "^0.10.2",
        "azure-search": "0.0.8",
        "cheerio": "^0.19.0",
        "documentdb": "^1.9.0",
        "feedparser": "^1.1.4",
        "google-timezone-api": "^1.0.1",
        "googlemaps": "^1.0.0",
        "iconv-lite": "^0.4.13",
        "lodash": "^2.4.1",
        "moment": "^2.10.0",
        "moment-timezone": "^0.4.0",
        "newrelic": "^1.26.1",
        "nodemailer": "^2.4.1",
        "nodemailer-smtp-transport": "^2.5.0",
        "password-generator": "^1.0.0",
        "pkginfo": "^0.3.0",
        "request": "^2.60.0",
        "socket.io": "^1.4.4",
        "underscore": "^1.8.3"
      },
      "devDependencies": {
        "arrow": "*",
        "grunt": "^0.4.5",
        "grunt-contrib-clean": "^0.6.0",
        "grunt-contrib-jshint": "^0.10.0",
        "grunt-kahvesi": "^0.1.0",
        "grunt-mocha-test": "^0.11.0",
        "mocha": "^1.21.4",
        "should": "^4.0.4"
      },
      "main": "app.js",
      "engines": {
        "node": "0.12.4"
      }
    }

Parijat sahai 2016-07-29

Btw, I find it really odd that issues with ArrowDB would affect registry server at Appcelerator - how lean are you running your operations that you can't have separate servers and redundancy and failover for ANY of your servers? Unbelievable and highly unprofessional I must say!
Ingo Muschenetz 2016-07-29

I disagree. We specifically run our infrastructure on the same infrastructure as our public customers as a point of pride. Customers are welcome to purchase a VPC for themselves if they like, but typically they do that for organizational reasons. We certainly could set one up for ourselves, but we specifically choose not to. We also have multiple servers for redundancy, but the issue here was separate and appears to be due to a bug in MongoDB code.
Ingo Muschenetz 2016-07-29

This is a NPM issue. You can get a similar thing to happen by just doing a npm install arrow --loglevel verbose However, there is still some work around figuring what is triggering it and how to stop it. We will work on that ASAP.
Parijat sahai 2016-07-29

If what you're saying is correct, then I have to assume that EVERY customer of Appcelerator has been impacted by this MongoDB bug. If there are any mission-critical applications being run by your customers, I would expect that either your customers would be fleeing from Appcelerator by now or screaming like crazy because it has been more than 4 days since the issue arose. If neither of that has happened, then it seems no customer trusts Appcelerator's public infrastructure enough to run their mission-critical applications and probably only run peripheral applications that go unnoticed when they are down for such an extended amount of time. Hence, Appcelerator cannot pride itself about its infrastructure being able to support mission-critical applications on their cloud. And blaming the outage on a third-party software doesn't help - it should have been tested by Appcelerator before deploying on their public cloud and, for the worst case scenario, Appcelerator should have done rollback planning in the case things go horribly wrong, which is what happened here. Clearly, contingency planning wasn't done and customers are now being asked to wait patiently during the 100+ hour outage - give me a break!
Ingo Muschenetz 2016-07-29

[~parijatsahai] I don't believe it's a fruitful exercise to go down this path. You are more than welcome to contact support if you have additional questions. My point initially was to say I believe we've addressed the first issue and not lead down a complete explanation of our operations practices. We take all of our customer's applications very seriously. I will only say that we have many enterprise customers both on public cloud and VPC. Once we were able to identify the issue, we immediately restored service to all public cloud users (VPC users are unaffected). I understand a percentage of users are affected by this issue related to the bug I mentioned (which is related to a very specific use case) and we are addressing those ASAP.
Ingo Muschenetz 2016-07-29

This ticket has been cloned and moved to API-1325 to see what we can do to address the issue.
Parijat sahai 2016-07-29

You said "we immediately restored service to all public cloud users". However, our application continues to get ArrowDB errors "503 Service Unavailable. No server is available to handle this request." - clearly the ArrowDB service hasn't been restored in over 4 days now. Thanks for acknowledging and reproducing the issue with the build. Hopefully we will have a quick resolution. I'm still at a loss to understand why only 2 customers have reported this issue and how I'm the first one to do that - it would impact either ALL or NONE of the customers - there probably aren't many Arrow customers, are there? If you put yourself in my shoes, you'd understand how scary this looks.
Ingo Muschenetz 2016-07-29

The build issue is intermittent. NPMJS has many, many registries. It does not seem to affect everyone all the time. You could have cached the dependencies in your .npm folder, meaning that you wouldn't redownload them. There are a lot of variables that affect this, which is why not everyone would encounter it. http://status.npmjs.org/
Sharif AbuDarda 2016-08-30

Hello, [~parijatsahai], are you still having the issue?
Parijat sahai 2016-08-31

The issue with timeout was resolved after we changed the Node engine version on the Arrow app's package.json file to 4.x.

JSON Source